JP2022538669A - Improved eye tracking latency - Google Patents

Abstract

Systems and methods for improving eye tracking latency. An exemplary head-mounted system acquires a first image of a user's eyes. The first image is provided as input to a machine learning model that has been trained to generate iris segmentation data and pupil segmentation data given an image of the eye. A second image of the eye is acquired. A set of locations in the second image where one or more flashes are shown are detected based on the iris segmentation data generated for the first image. A region of the second image where the pupil of the user's eye is shown is identified based on the pupil segmentation data generated for the first image. A user's eye pose is determined based on the detected set of flash locations in the second image and the identified regions of the second image.

Description

(included by reference)
This application is made by reference to the following patent applications: U.S. Provisional Patent Application No. 62/871,009, filed July 5, 2019; Incorporates the entirety of each of US Provisional Patent Application No. 62/945,562.

The present disclosure relates generally to processing eye images, and more specifically to systems and methods for estimating a detailed eye shape model comprising a pupil, iris, or eyelids using cascaded shape regression. Regarding. The human iris can be used as a source of biometric information. Biometric information can provide authentication or identification of an individual. Biometric information can additionally or alternatively be used to determine eye gaze direction.

Systems and methods for robust biometric applications using detailed eye shape models are described. In one aspect, after receiving an eye image of the eye (eg, from an eye tracking camera on an augmented reality display device), the eye shape of the eye (eg, upper or lower eyelid, iris, or pupil shape) in the eye image. is calculated using the cascaded shape regression method. Eye features associated with the estimated eye shape can then be determined and used in biometric applications such as gaze estimation or biometric identification or authentication (eg, iris code). A cascaded shape regression method can be trained on a set of annotated eye images that label the shape of the eyelid, pupil, and iris, for example.

Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will be apparent from the description, drawings, and claims. Neither the Summary of the Invention nor the following Detailed Description is meant to define or limit the scope of the present subject matter.

FIG. 1A schematically illustrates an example of an eye showing exemplary eye features.

FIG. 1B shows an example of three angles (eg, yaw, pitch, and roll) that can be used to measure eye pose direction relative to the eye's natural resting state.

FIG. 2A schematically illustrates an embodiment of a wearable display system.

FIG. 2B schematically illustrates a top view of an embodiment of a wearable display system.

FIG. 3 is a flow diagram of an exemplary routine for extracting biometric information from eye images used in biometric applications.

FIG. 4A graphically illustrates an exemplary progression of detailed eye shape model estimation.

FIG. 4B graphically illustrates an example of a detailed eye shape model in which pupil, iris, and eyelid boundaries have been identified.

FIG. 4C is an image showing an example of two pairs of shape indexing features.

FIG. 5 illustrates an example of a set of annotated training images used to learn the regression function.

FIG. 6 is a flow diagram of an embodiment of an eye shape training routine for learning cascaded shape regression.

FIG. 7A diagrammatically illustrates an example of an erroneous boundary point.

FIG. 7B diagrammatically illustrates an example of selective feature detection.

FIG. 8 is a schematic diagram of a wearable system including an eye tracking system.

FIG. 9 is a block diagram of a wearable system that may include an eye tracking system.

FIG. 10 is a block diagram of a wearable system that may include an eye tracking system.

FIG. 11 is a flow chart illustrating an exemplary process for performing eye tracking with reduced latency.

Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of the disclosure.

DETAILED DESCRIPTION Overview Extracting biometric information from the eye generally involves a procedure for segmenting the iris within the eye image. Iris segmentation includes the steps of locating the iris boundary, finding the pupillary and limbal boundaries of the iris, locating the upper or lower eyelid if they occlude the iris, locating eyelashes, shadows, or reflections. Actions can be involved, including detecting and filtering out occlusion, and the like. For example, the eye image may be contained within an image of the face, or may be an image of the periocular region. To perform iris segmentation, both the boundaries of the pupil (e.g., the inner boundary of the iris) and the limbus (e.g., the outer boundary of the iris) can be identified as separate segments of the image data. In addition to this segmentation of the iris, the part of the iris that is occluded by the (upper or lower) eyelid can be estimated. This estimate is made because the entire iris of an individual is rarely visible during normal human activity. For example, the entire iris may generally be immune to eyelid occlusion (eg, during blinking).

The eyelid can be used by the eye to keep the eye moist, for example, by spreading tears and other secretions across the ocular surface. Eyelids may also be used to protect the eye from foreign debris. As an example, the blink reflex protects the eye from acute trauma. As another example, the eyelid may protect the eye even when the eye is actively viewing the world, for example, by automatically moving in response to changes in the eye's pointing direction. Such movement by the eyelid can maximize ocular surface protection while avoiding pupillary occlusion. However, this transfer presents additional challenges when extracting biometric information with iris-based biometric measurements such as iris segmentation. For example, to use iris segmentation, the area of the iris that is occluded by the eyelid can be estimated and hidden from the identification calculation, or images taken during the eyelid blink are discarded during analysis. , or may be given a lower weight.

Extracting biometric information presents challenges such as estimating the portion of the iris that is occluded by the eyelid. However, using the techniques described herein, the challenges presented in extracting biometric information can be mitigated. For example, the problem can be at least partially alleviated by first estimating the eye shape. As used herein, eye shape includes one or more of pupil, iris, upper eyelid, or lower eyelid shapes (eg, boundaries). This estimate of eye shape may be used as a starting point for iris segmentation in some embodiments.

Once the eye shape is estimated, biometric applications can be performed more efficiently and more robustly. For example, corneal reflections (eg, glints) found in certain regions of the eye (eg, iris) can be used for gaze estimation. Flashes in other regions of the eye (eg, the sclera) may not be used in eye gaze estimation in some embodiments. By computing a detailed eye shape model using the techniques described herein, flashes in the desired region (e.g. iris) search the entire eye (e.g. iris and sclera) It can be located more quickly and efficiently by removing the need and thus producing a more efficient and robust line-of-sight estimation.

Algorithms exist for tracking eye movements of computer users to obtain biometric information. For example, a camera coupled to a computer monitor can provide images for identifying eye movements. However, the cameras used for eye tracking are some distance from the user's eyes. For example, a camera may be placed on top of a user's monitor that is coupled to a computer. As a result, the images of the eye produced by the camera are often produced at different angles with poor resolution. Therefore, extracting biometric information from captured eye images can present challenges.

In the context of wearable head-mounted displays (HMDs), the camera may advantageously be closer to the user's eyes than the camera coupled to the user's monitor. For example, the camera may be mounted on a wearable HMD, which itself is placed on the user's head. Proximity of the eye to such a camera can result in higher resolution eye images. Thus, computer vision techniques are capable of extracting visual features from a user's eye, particularly the iris (eg, iris features) or the sclera surrounding the iris (eg, sclera features). For example, the iris of the eye will show detailed structure when viewed by a camera near the eye. Such iris features can be particularly noticeable when viewed under infrared (IR) illumination and can be used for biometric applications such as gaze estimation or biometric identification. These iris features are unique for each user and can be used to uniquely identify the user in the manner of a fingerprint. Ocular features can include blood vessels within the sclera (outside of the iris) of the eye, which can also be particularly noticeable when viewed under red or infrared light. Eye features may further include glare and pupil center.

With the techniques disclosed herein, detailed eye shape estimation is used to generate more robust techniques for detecting eye features used in biometric applications (e.g., gaze estimation and biometric identification). may be used. The use of gaze estimation has a significant impact on the future of computer interfaces. Gaze estimation is currently employed in active interfaces (eg, interfaces that receive commands through eye movement) and passive interfaces (eg, virtual reality devices that modify the display based on gaze position). Detecting eye features using conventional eye shape estimation techniques is difficult due to image noise, ambient light, and large variations in appearance when the eye is half-closed or blinked. . Therefore, techniques that generate more robust algorithms for determining eye characteristics for use in biometric applications such as gaze estimation or biometric identification would be advantageous. The following disclosure describes such methods.

This disclosure will describe a detailed eye shape model computed using a cascaded shape regression technique and how the detailed eye shape model can be used for robust biometric applications. In recent years, shape regression has become a state-of-the-art approach for accurate and efficient shape matching. It has been successfully used for face, hand, and ear shape estimation. Regression techniques can, for example, capture large differences in appearance, enforce shape constraints between landmarks (e.g., iris between eyelids, pupil inside iris), and are computationally efficient. Therefore, it is advantageous. Although regression techniques are described, it should be understood that neural networks may be employed as an alternative to and/or in combination with regression techniques. For example, non-linear combinations of regressions may be utilized and fall within the scope of the disclosure herein.

As used herein, video may include, but is not limited to, recordings of sequences of visual images. Each image in a video is sometimes referred to as an image frame or frame. A video can include multiple consecutive or non-consecutive frames with or without an audio channel. A video may include multiple frames that are ordered in time or unordered in time. Hence, the images in the video may be referred to as eye image frames or eye images.

Examples of Eye Images FIG. Curve 114a shows the pupil boundary between pupil 114 and iris 112, and curve 112a shows the limbus boundary between iris 112 and sclera 113 (the "white eye" of the eye). The eyelid 110 includes an upper eyelid 110a, a lower eyelid 110b, and eyelashes 117. As shown in FIG. Eye 100 is illustrated in a natural resting position. For example, a static pose may represent a pose in which both the user's face and line of sight are oriented such that they would be directed toward a distant object in front of the user. The natural resting posture of the eye 100 can be indicated by the natural resting direction 180, which is orthogonal to the surface of the eye 100 when in the natural resting posture (e.g., directly from the plane of the eye 100 shown in FIG. 1A). outward) direction, centered within the pupil 114 in this example.

Eye 100 may include eye features 115 in the iris or sclera (or both) that may be used for biometric applications such as eye tracking. FIG. 1A illustrates an example of ocular features 115, including iris features 115a and scleral features 115b. Eye features 115 may be referred to as individual feature points. Such ocular features 115 may be unique to an individual's eye and may be distinctly different for each eye of that individual. Iris feature 115a may be a point of particular color density relative to the rest of the iris color or relative to some area surrounding the point. As another example, an iris texture (eg, a texture that differs from the iris texture in the vicinity of the feature) or pattern can be identified as an iris feature 115a. As yet another example, iris feature 115 a may be a scar that differs in appearance from iris 112 .

Eye features 115 can also be associated with the blood vessels of the eye. For example, blood vessels may exist outside the iris 112 but within the sclera 113 . Such vessels may be more prominently visible under red or infrared light illumination. Scleral feature 115b may be a blood vessel within the sclera of the eye.

Additionally or alternatively, eye features 115 may include flashes of light, including corneal reflections of light sources (eg, IR light sources directed toward the eye for eye tracking or biometric identification). In some cases, the term "ocular feature" refers to a feature of the eye, whether the feature is in the iris 112, in the sclera 113, or seen through the pupil 114 (e.g., on the retina). It may be used to refer to any type of identifying feature in or above.

Each eye feature 115 can be associated with a descriptor, which can be a numerical representation of the area surrounding the eye feature 115 . A descriptor may also be referred to as an iris feature representation. As yet another example, such ocular features include scale-invariant feature transform (SIFT), accelerated robust features (SURF), features from accelerated segment testing (FAST), oriented FAST and rotational brief (ORB), It can be derived from KAZE, accelerated KAZE (AKAZE), and so on.

Thus, eye features 115 may be derived from known algorithms and techniques from the field of computer vision. Such eye features 115 may be referred to as feature points. In some of the exemplary embodiments described below, eye features will be described in terms of iris features. This is not a limitation, and any type of ocular feature (eg, scleral features) can additionally or alternatively be used in other implementations.

As the eye 100 moves to look toward different objects, the line of sight of the eye (also sometimes referred to herein as eye pose) will change relative to the natural resting direction 180 . The current eye gaze direction can be measured relative to the natural resting eye gaze direction 180 . The current line of sight of the eye 100 can be expressed as three angular parameters that indicate the current eye pose direction relative to the eye's natural resting direction 180 . For purposes of illustration, and with reference to the exemplary coordinate system shown in FIG. 1B, these angular parameters are α (which may be referred to as yaw), β (which may be referred to as pitch), and γ (which may be referred to as roll). obtained). In other implementations, other techniques or angular representations for measuring eye line-of-sight may be used, such as any other type of Euler angle system.

Eye images can be obtained from the video using any suitable process. For example, images may be extracted from one or more consecutive frames. Eye pose can be determined from eye images using various eye tracking techniques. For example, eye pose may be determined by considering the lens effect of the cornea on the light source provided, or by calculating the shape of the pupil or iris (for a circle representing a forward-looking eye). can be done.

Example Wearable Display System Using Eye Shape Estimation In some embodiments, the display system may be wearable, which advantageously allows for more immersive virtual reality (VR), augmented reality (AR) ), or provide a mixed reality (MR) experience, in which digitally reproduced images or portions thereof are presented to the wearer in a manner that makes them appear or be perceived as real.

Without being limited by theory, it is believed that the human eye can typically interpret a finite number of depth planes to provide depth perception. As a result, a highly believable simulation of perceived depth can be achieved by presenting the eye with a different representation of the image corresponding to each of these limited number of depth planes. For example, a display containing a stack of waveguides may be configured to be worn positioned in front of a user's or viewer's eye. A stack of waveguides uses multiple waveguides to transmit image information from an image input device (e.g., a discrete display or output end of a multiplexed display that transmits image information via one or more optical fibers) to a specific By directing light to the viewer's eye at a specific angle (and divergence) corresponding to the depth plane associated with the waveguide, it can be utilized to provide the eye/brain with three-dimensional perception.

In some embodiments, two stacks of waveguides (one for each eye of the viewer) may be utilized to provide different images to each eye. As an example, an augmented reality scene may be such that an AR technology wearer sees a real-world park-like setting featuring people, trees, buildings in the background, and a concrete platform. In addition to these items, wearers of AR technology can also "see" a robot statue standing on a real-world platform and a flying cartoon-like avatar character that appears to be an anthropomorphic bumblebee. Perceived, but the image of the robot and the bumblebee do not exist in the real world. A stack of waveguides can be used to generate a light field corresponding to an input image, and in some implementations the wearable display comprises a wearable light field display. An example of a wearable display device and waveguide stack for providing light field images is incorporated herein by reference in its entirety with respect to everything it contains. ).

2A and 2B illustrate an example wearable display system 200 that can be used to present a VR, AR, or MR experience to a wearer 204. FIG. Wearable display system 200 can be programmed to capture images of the eye and perform eye shape estimation to provide any of the applications or embodiments described herein. Display system 200 includes a display 208 (eg, which can be positioned in front of one or both eyes of a user) and various mechanical and electronic modules and systems to support the functionality of display 208. . The display 208 may be coupled to a frame 212 , which is wearable by the display system wearer or viewer 204 and configured to position the display 208 in front of the wearer's 204 eyes. Display 208 may be a light field display configured to display virtual images at multiple depth planes from the user. In some embodiments, speaker 216 is coupled to frame 212 and positioned adjacent the user's ear canal, and in some embodiments another speaker, not shown, is adjacent the user's other ear canal. Positioned and provides stereo/shapable sound control. The display 208 is operably coupled 220 , such as by wired leads or wireless connectivity, to a local data processing module 224 , which is fixedly attached to a helmet or cap worn by the user, which is fixedly attached to the frame 212 . It may be mounted in a variety of configurations, such as attached to the device, embedded in headphones, or otherwise removably attached to the user 204 (eg, in a backpack configuration, in a belt-tie configuration).

As shown in FIG. 2B, wearable display system 200 may further include an eye tracking camera 252a disposed within wearable display system 200 and configured to capture an image of eye 100a. Display system 200 may further comprise light source 248a configured to provide sufficient illumination to capture ocular features 115 of eye 100a using eye-tracking camera 252a. In some embodiments, light source 248a illuminates eye 100a using infrared light that is not visible to the user so that the user is not distracted by the light source. Eye tracking camera 252 a and light source 248 a may be separate components that are separately attached to wearable display system 200 . For example, the components may be attached to frame 212 . In other embodiments, eye-tracking camera 252a and light source 248a may be components of a single housing 244a that is attached to frame 212 . In some embodiments, wearable display system 200 may further comprise a second eye-tracking camera 252b and a second light source 248b configured to illuminate and capture an image of eye 100b. . Eye tracking cameras 252a, 252b can be used to capture eye images for use in eye shape calculation, line of sight determination, and biometric identification.

Referring again to FIG. 2A, the local processing and data module 224 may comprise a hardware processor and non-volatile memory, e.g., non-transitory digital memory such as flash memory, both of which are used to process data, cache , and can be used to aid memory. The data may be (a) an image capture device (such as a camera), microphone, inertial measurement unit, accelerometer, compass, GPS unit, wireless device, and/or gyroscope, etc. (e.g., operably coupled to frame 212). , or otherwise attached to the wearer 204), and/or possibly (b) the remote processing module 228 and/or and data obtained and/or processed using remote data repository 232 . The local processing and data module 224 may be operably coupled to a remote processing module 228 and a remote data repository 232 by communication links 236, 240, such as via wired or wireless communication links, such that these remote modules 228, 232 are operably coupled to each other and available as resources to the local processing and data module 224 .

In some embodiments, remote processing module 228 may comprise one or more processors configured to analyze and process data and/or image information, such as video information captured by image capture devices. Video data may be stored locally within local processing and data module 224 and/or within remote data repository 232 . In some embodiments, remote data repository 232 may comprise a digital data storage facility, which may be available through the Internet or other networking configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 224, allowing fully autonomous use from remote modules. In some implementations, the local processing and data module 224 and/or the remote processing module 228 are programmed to perform embodiments of estimating detailed eye shape models as described herein. . For example, local processing and data module 224 or remote processing module 228 can be programmed to implement an embodiment of routine 300 described with reference to FIG. 3 below. Local processing and data module 224 or remote processing module 228 may use the eye shape estimation techniques disclosed herein to perform biometric applications, for example, to identify or authenticate the identity of wearer 204. , can be programmed. Additionally or alternatively, gaze estimation or pose determination, for example, determines the direction each eye is looking toward.

The image capture device can capture video for specific applications (eg, video of the wearer's eyes for eye tracking applications or video of the wearer's hands or fingers for gesture identification applications). The video can be analyzed by one or both of the processing modules 224, 228 using eye shape estimation techniques. In this analysis, processing modules 224, 228 can perform eye shape estimation for robust biometric applications. As an example, the local processing and data module 224 and/or the remote processing module 228 can be programmed to store eye images acquired from eye-tracking cameras 252 a , 252 b mounted on the frame 212 . In addition, the local processing and data module 224 and/or the remote processing module 228 process eye images using the eye shape estimation techniques (eg, routine 300) described herein to provide wearable display system 200 can be programmed to extract biometric information of the wearer 204 of the. In some cases, offloading at least some of the biometric information (eg, into the "cloud") to a remote processing module may improve efficiency or speed of computation. Various parameters related to eye gaze identification (eg, weights, bias terms, random subset sampling factors, numbers, sizes of filters (eg, Sobel differential operators), etc.) can be stored in data modules 224 or 228. .

The results of video analysis (eg, detailed eye shape models) can be used by one or both of processing modules 224, 228 for additional actions or processing. For example, biometric identification, eye tracking, recognition, or classification of objects, poses, etc. may be used by wearable display system 200 in various applications. For example, video of the wearer's eyes can be used for eye shape estimation, which in turn determines the direction of the wearer's 204 gaze through the display 208, using the processing module 224, 228 can be used. The processing modules 224, 228 of the wearable display system 200 can be programmed with one or more embodiments of eye shape estimation to perform any of the video or image processing applications described herein. .

Exemplary Eye Shape Estimation Routine FIG. 3 is a flow diagram of an exemplary eye shape estimation routine 300 . Eye shape estimation routine 300 can be implemented by local processing and data module 224 or remote processing module 228 and data repository 232, described with reference to FIG. Eye shape estimation may also be referred to as eye shape detection or detailed eye shape modeling.

Routine 300 begins at block 308 when eye image 324 is received. The eye image 324 may be received from a variety of sources including, for example, an image capture device, head mounted display system, server, non-transitory computer readable medium, or client computing device (eg, smart phone). can be done. Eye image 324 may optionally be received from eye-tracking camera 252a. In some implementations, the eye image 324 can be extracted from a video (eg, an eye video).

At block 312 a detailed eye shape model 400 b may be estimated from the eye image 324 . In some embodiments, detailed eye shape model 400b may be estimated using cascaded shape regression, as further described below.

At block 316 , eye features 115 are determined based at least in part on the detailed eye shape model 400 b estimated at block 312 . In some embodiments, the eye features 115 (some of which are shown in image 332) include the pupil or limbus boundary, the eyelid boundary, the glint, the eye feature point, or the center of the pupil 114. Eye features 115 may also include any feature that can be used in biometric applications. The detailed eye shape model 400b estimated at block 312 may serve as prior knowledge and improve robustness of feature detection at block 316 .

At block 320 , biometric applications (eg, gaze estimation or biometric identification/authentication) are performed based at least in part on the biometric information obtained at blocks 312 and 316 . In some embodiments, at block 320 a the gaze direction may be estimated based at least in part on the eye features 115 determined at block 316 . Additionally or alternatively, in some embodiments, at block 320 b biometric identification/authentication may be performed based at least in part on the eye characteristics determined at block 316 . Biometric identification or authentication may include determining an iris code (eg, an iris code based on the Daugman algorithm) based at least in part on the eye image and the determined pupil and limbus boundaries.

ｆ_ｔは、段階ｔにおける回帰関数であり、Φ_ｔは、形状インデックス化抽出関数である。Φ_ｔは、前の段階Ｓ_ｔ－１における入力画像Ｉおよび形状の両方に依存し得ることに留意されたい。形状インデックス化抽出関数Φ_ｔは、「非形状インデックス化」特徴と比較して、より大きい形状変動に対処することができる。大域的照明変化に不変であり得る、一対ピクセル比較特徴が、使用されてもよい。回帰は、形状インクリメントΔＳ_ｔを前の段階Ｓ_ｔ－１における形状に加算し、Ｓ_ｔ＝Ｓ_ｔ－１＋ΔＳ_ｔを生じさせることによって、次の段階ｔ＋１に進む。 Exemplary Eye Shape Estimation Considering an input image I with an initial eye shape S ₀ , cascaded shape regression progressively refines the shape S by estimating shape increments ΔS in steps. The initial shape S ₀ may represent a best guess to an eye shape (eg, pupil, limbus, and eyelid border) or a default shape (eg, circular pupil and iris border centered on eye image I). In general form, the shape increment ΔS _t at stage t is regressed as

f _t is the regression function at stage t and Φ _t is the shape-indexed extraction function. Note that Φ _t can depend on both the input image I and the shape in the previous stage S _t-1 . The shape-indexed extraction function Φ _t can handle larger shape variations compared to the 'non-shape-indexed' features. A pairwise pixel comparison feature may be used that may be invariant to global lighting changes. The regression proceeds to the next stage t+1 by adding the shape increment ΔS _t to the shape in the previous stage S _t−1 yielding S _t =S _t−1 +ΔS _t .

Some examples of cascaded shape regression models that can be used to estimate eye shape are explicit shape regression (ESR), cascaded postural regression (CPR), linked regression trees (ERT), supervised descent Method (SDM), Local Binary Features (LBF), Probabilistic Random Forest (PRF), Cascaded Gaussian Process Regression Trees (cGPRT), Coarse-Fixed Shape Search (CFSS), Random Cascading Regression Tree Swarm (R-CR-C) , cascaded collaborative regression (CCR), spatio-temporal cascaded shape regression (STCSR), or other cascaded shape regression methods.

FIG. 4A diagrammatically illustrates an exemplary progression of a detailed eye shape model. For simplicity, FIG. 4A depicts only the shape of the upper and lower eyelids 110a, 110b and does not show the estimated shape of the iris 112 or pupil 114 as shown in FIG. However, the shape of the iris 112 and pupil 114 may additionally or alternatively be modeled at this stage (see, eg, exemplary results in FIG. 4B). In some embodiments, initial estimated eye shape 404 may be any eye shape similar to target shape 412 . For example, the initial eye shape estimate can be set as the average shape at the center of the image.

FIG. 4A depicts eye shape regression from an initial estimated eye shape 404 to a target shape 412 performed over 11 steps. For example, an initial (zeroth) stage S ₀ , a first stage S ₁ , and a tenth stage S ₁₀ are shown. For simplicity, only the intermediate eyelid shape 408 is depicted in FIG. 4A. In some embodiments, the regression model may be programmed to stop after a predetermined number of iterations (eg, 5, 10, 20, 50, 100, or more). In other embodiments, the regression model may continue to iterate until the shape increment ΔS _t at stage t is less than a threshold. For example, if the relative eye shape change |ΔS _t /S _t | is less than a threshold (eg, 10 ⁻² , 10 ⁻³ , or less), the regression model may terminate. In other embodiments, the regression model may continue to iterate until the difference between shape S t at stage _t and shape S _t−1 at the previous stage is less than a threshold.

In some embodiments, the detailed eye shape model 400b may comprise multiple boundary points 424 for the pupil, limbus, or lid boundary. Boundary points 424 may correspond to estimated eyelid shape 412, estimated iris shape 416, and estimated pupil shape 420, respectively. The number of demarcation points 424 can be in the range of 6-100 or more. In some implementations, the detailed eye shape model 400b can be used to determine whether a received eye image meets certain standards, eg, image quality.

FIG. 4B illustrates an example of a completed eye shape model. This model may be determined using the eye shape estimation routine described in FIG. For example, the completed eye shape model may represent the result of block 312 after the eye shape has been modeled based on cascaded shape regression that determined the pupil, limbus, and eyelid boundaries. These boundaries are overlaid on an image of the periocular region of the eye to show the match between the computed boundaries and the underlying eye image. As explained above, the shape-indexed extraction function Φ _t can accommodate greater shape variation compared to the 'non-shape-indexed' features. A pairwise pixel comparison feature may be used that may be invariant to global lighting changes.

FIG. 4C is an image showing an example of two pairs of shape indexing features (eg, 460a, 460b). A local coordinate system (shown as x and y axes 450) is determined by the current eye shape (eg, eyelid shape 462). Intensity values from a pair of pixel locations 460a, 460b (squares connected by arrowed lines, two pairs 460a, 460b of such pixel locations are shown) are used to measure binary features (e.g., match or mismatch). can be compared to provide a Boolean value such as 0 or 1) indicating a match. For example, pixels located inside the pupil (eg, pupil pixels in feature 460b) are located outside the pupil (eg, within the user's iris, sclera, or skin (as shown in FIG. 4C)). ) may be darker in color or contrast than the pixel in which it is located.

In some implementations, the pixel locations may be fixed within the local coordinate system 450, which varies as the eye shape 462 is updated during the regression phase. In one exemplary system, 2,500 features are constructed from 400 pixel locations, which may be learned from training data. Learning from training data is described in more detail below with respect to FIG.

Examples of Training Images for Learning Cascaded Shape Regression In some embodiments, the regression function f _t and the shape indexing extraction function Φ _t described above are used for annotated (e.g., labeled) training data. can be learned from a set.

FIG. 5 illustrates an example of training data 500 including eight exemplary eye images from different subjects with large shape and appearance variations (indexed as (a) to (h)). Targeted eye images advantageously include a wide range of eye variations (e.g., normally open eyes, blinking eyes, natural rest orientations) from a wide range of subjects (of different genders, ethnicities, etc.). direction (such as an eye pointing up, down, left, right)).

The training data 500 is annotated to indicate the features to be learned. In the illustrated example, these features include the pupil, limbus, and eyelid border marked on each of the images. These landmark boundaries within each of the images in training data 500 can be determined using any suitable pupil, limbus, or eyelid boundary technique, or manually.

Various machine learning algorithms may be used to learn the regression function f _t and the shape-indexed extraction function Φ _t from the annotated training data 500 . Supervised machine learning algorithms (eg, regression-based algorithms) can be used to learn regression functions and shape-indexed extraction functions from the annotated data 500 . Some examples of machine learning algorithms that can be used to generate such models include regression algorithms (e.g., least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision Tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., a priori algorithms, etc.), artificial neural networks algorithms (such as perceptrons), deep learning algorithms (such as deep Boltzmann machines or deep neural networks), dimension reduction algorithms (such as principal component analysis), ensemble algorithms (such as stack generalization), or other May include machine learning algorithms.

In some embodiments, a set of training images may be stored in remote data repository 232 illustrated in FIG. 2A. Remote processing module 228 may access training images and learn regression function f _t and shape-indexed extraction function Φ _t . Local processing and data module 224 may then store regression function f _t and shape-indexed extraction function Φ _t on wearable device 200 . This reduces the need for the local processing and data module 224 to perform the computationally intensive process of learning the regression function f _t and the shape indexing extraction function Φ _t .

In some embodiments, biometric information may be taken from user 204 and stored on local processing and data module 224 . The biometric information is then used by the local processing and data module 224 (or can be used by the remote processing module 228). Such training more specifically personalizes the regression model so that it matches the characteristics of the user's eye and periocular region and can improve accuracy and efficiency.
Exemplary Eye Shape Training Routine

FIG. 6 is a flow diagram of an exemplary eye shape training routine 600 that may be used to learn the regression function f _t and the shape indexed extraction function Φ _t . For example, a function can be learned based on a set of training images (eg, images 500 shown in FIG. 5). Eye shape training routine 600 may be implemented by processing modules 224, 228, 232 and/or one or more other processors.

Routine 600 begins at block 608 when training data (eg, data 500) comprising annotated eye images is accessed. Training data can be accessed from a non-transient data store that stores annotated eye images. Processing modules can access non-transitory data storage via wired or wireless techniques.

At block 612, machine learning techniques (eg, supervised learning for annotated or labeled images) are applied to learn the regression function f _t and the shape-indexed extraction function Φ _t . A cascaded shape regression model can then be generated at block 616 . This regression model allows routine 300 to estimate a detailed eye shape model at block 312 . As explained above, the cascaded shape regression model is adapted to a particular user by further training the regression function and the shape indexed extraction function on the user's eye images acquired by the wearable display system 200 during use. can be personalized.

Example of Robust Feature Detection Eyelid Occlusion of Pupil or Iris FIG. 7A illustrates a boundary point 424 of the pupil that is partially occluded by the eyelid. In one embodiment for robust feature detection using a detailed eye shape model, pupil detection may be improved by removing erroneous pupil boundary points 704 . Exemplary point 704 is shown as an arc of boundary points within pupillary boundary 420 along upper eyelid 110a.

A false pupil boundary point 704 can be created when the eyelid partially occludes the pupil, as shown in FIG. 7A (upper eyelid 110a partially occludes pupil 114). Point 704 therefore reflects the position of the eyelid rather than the true boundary of the pupil (which is occluded by the eyelid). Rather than including erroneous boundary points 704, which can lead to the generation of an inaccurate model of the pupil, the erroneous boundary points 704 may be identified and removed before the pupil boundary finding method is implemented. In some embodiments, the false pupillary boundary point 704 may be any pupillary boundary point located within a certain distance of the upper or lower eyelid. In some embodiments, the false pupil demarcation point 704 may be any pupil demarcation point adjacent to the upper or lower eyelid.

In some embodiments, once the erroneous pupil boundary points 704 are identified and removed, an ellipse may be fitted to the pupil using the remaining pupil boundary points. Algorithms that may be implemented for such elliptic fitting include calculus operators, least squares methods, random sample consensus (RANSAC), or elliptic or curve fitting algorithms.

Note that although the above embodiments specifically refer to erroneous pupil boundary points, the techniques described above can also be applied to identify and remove erroneous limbal boundary points. .

In some embodiments, a detailed eye shape model may be used in conjunction with a pupil boundary finding algorithm, such as the starburst algorithm, which may be employed, for example, to detect many pupil boundary points. Using the eyelid shape 412 of the detailed eye shape model, the boundary points determined using the starburst algorithm adjacent to the upper or lower eyelids 110a, 110b are removed and the remaining boundary points are the pupil boundary 420 used to conform to In some embodiments, the limbus boundary point adjacent to the sclera 113 may also be identified using the detailed eye shape model. An iris ellipse 416 is then fitted using only the limbus boundary points determined to be adjacent to the sclera 113 . Similarly, pupil boundary 420 may be fitted using only pupil boundary points determined to be adjacent to iris 112 . In some embodiments, the detailed eye shape model may improve the robustness of the pupil boundary finding algorithm by providing a better initial "best guess" of the pupil center based on the detailed eye shape model. .

Flash Detection In conventional gaze estimation, pupil boundaries (eg, ellipses in some techniques) and glints are detected by searching the entire eye image. Given the detailed eye shape model described herein, feature detection can be faster and more efficient by eliminating the need to search the entire eye for features. In some embodiments, by first identifying different regions of the eye (e.g., sclera, pupil, or iris), the detailed eye shape model is used for feature detection (e.g., selection) in specific regions of the eye. feature detection).

FIG. 7B illustrates an example of selective feature detection. Flashes 115 a , 115 b may appear in sclera 113 , iris 112 , or pupil 114 . Flashes 115a, 115b may represent reflections from one or more light sources (eg, LEDs such as infrared LEDs). In some biometric applications, it is necessary to identify flashes in certain regions of the eye (eg, the iris, which represents corneal reflection), while ignoring flashes outside these regions (eg, the sclera), or can be desirable. For example, when determining the line of sight in one technique, scleral flashes 115b located within the sclera 113 may not represent reflections of the light source from the cornea, and their inclusion in the line of sight technique may result in an estimated line of sight uncertainty. leading to accuracy. Therefore, it may be advantageous to search and identify iris flashes 115a located within the iris 112 or within the limbus boundary 416 using a detailed eye shape model.

As illustrated in FIG. 7B, the iris flash 115a is within the iris 112 and may therefore be preferred for gaze estimation. In contrast, scleral flashes 115b appear within the sclera 113 and therefore may not be favorable for line-of-sight estimation. Thus, embodiments of the techniques disclosed herein can be used to identify eye regions where flashes are likely to occur, and eye regions outside of these regions are searched. , which improves the accuracy, speed, and efficiency of the technique.

Blink Detection In some embodiments, feature detection can be more robust and efficient by using a detailed eye shape model and determining whether a received eye image meets a certain quality threshold. . For example, detailed eye shape models are sufficient for estimating eye-confident eye shapes and for feature extraction and for performing biometric applications (e.g., gaze finding or biometric authentication/identification). May be used to determine if it is open.

In some embodiments, eye images may be discarded based on one or more quality measures. For example, if the distance between upper eyelid 110a and lower eyelid 110b is less than a threshold, the eye image is considered unusable and discarded, thus no features are extracted for biometric applications. In some embodiments, an eye image may be rejected if the upper eyelid 110a and lower eyelid 110b are separated by 5 mm or less. In another embodiment, the eye image is more than a percentage (eg, greater than 40%, 50%, 60%, 75%, or more) of the pupil 114 or iris 112 of the eyelids 110a, 110b. can be rejected if it is occluded by one or more of In another embodiment, an eye image may be rejected if some pupil boundary points 704 are adjacent to the upper eyelid 110a or the lower eyelid 110b. For example, if approximately half of the pupil boundary point 704 is adjacent to the eyelids 110a, 110b, it can be concluded that approximately half of the pupil 114 is occluded by the eyelids and thus the eye image is unsuitable for biometric applications. can be

In other embodiments, rather than rejecting and discarding the eye images, the eye images are used in biometric applications where there is less occlusion of the eye (e.g., between the upper eyelid 110a and the lower eyelid 110b). images whose distance is above the threshold) are assigned lower weights.

Example Eye Tracking System FIG. 8 illustrates a schematic diagram of a wearable system 800 that includes an eye tracking system. Wearable system 800 may, in at least some embodiments, include components located within head-mounted unit 802 and components located within non-head-mounted unit 804 . Non-head-mounted unit 804 may be, by way of example, a belt-mounted component, a handheld component, a component in a rucksack, a remote component, and the like. Incorporating some of the components of wearable system 800 into non-head-mounted unit 804 can help reduce the size, weight, complexity, and cost of head-mounted unit 802 .

In some implementations, some or all of the functionality described as being performed by one or more components of head-mounted unit 802 and/or non-head-mounted unit 804 may be implemented within wearable system 800. may be provided using one or more components included anywhere in the For example, some or all of the functionality described below associated with the processor (eg, CPU 812) of head-mounted unit 802 may be provided using the processor (eg, CPU 816) of non-head-mounted unit 804. may be used and vice versa.

In some embodiments, some or all of such functionality may be provided using peripheral devices of wearable system 800 . Further, in some implementations some or all of such functionality is provided to one or more cloud computing devices or other remote computing devices in a manner similar to that described above with reference to FIG. may be provided using a computing device located at

As shown in FIG. 8, wearable system 800 can include an eye tracking system, including camera 252, that captures an image of the user's eye 810. As shown in FIG. If desired, the eye tracking system may also include light sources 848a and 848b (such as light emitting diodes "LEDs"). Light sources 848a and 848b may produce flashes of light (eg, reflections from the user's eyes that appear in images of the eyes captured by camera 252). The positions of light sources 848a and 848b relative to camera 252 may be known, and consequently the position of the flashes within the image captured by camera 252 may be used in tracking the user's eyes.

In at least one embodiment, there may be one light source 848 and one camera 252 associated with one of the user's eyes 810 . In another embodiment, there is one light source 848 and one camera 252 associated with each of the user's eyes 810, as in the example described above with reference to FIG. 2A. good too. In still other embodiments, there may be one or more cameras 252 and one or more light sources 848 associated with one or each of the user's eyes 810 . As a specific example, there may be two light sources 848 a and 848 b and one or more cameras 252 associated with each of the user's eyes 810 . As another example, there may be three or more light sources, such as light sources 848a and 848b, and one or more cameras 252, associated with each of the user's eyes 810. FIG.

Eye tracking module 814 may receive images from eye tracking camera 252, analyze the images, and extract various information. As an example, eye tracking module 814 may determine the user's eye pose, the three-dimensional position of the user's eyes relative to eye tracking camera 252 (and head-mounted unit 802), and whether one or both of user's eyes 810 are in focus. direction, user's convergence-divergence motion depth (e.g., depth from user where user is focused), user's pupil position, user's cornea and corneal sphere position, respective rotation of user's eye The center and/or line-of-sight center of each of the user's eyes may be detected.

Eye tracking module 814 may use techniques described below to extract such information. Additional systems and techniques for extracting and using eye tracking information are disclosed in U.S. Patent Application Serial Nos. 16/250,931 and 16/251,017, both filed Jan. 17, 2019; are expressly incorporated herein by reference in their entirety for all purposes). In some implementations, one or more such systems and techniques for extracting and using eye tracking information are part of one or more of the systems and techniques described herein, or in conjunction therewith may be employed. As shown in FIG. 8, eye tracking module 814 may be a software module implemented using CPU 812 within head-mounted unit 802 .

Data from the eye tracking module 814 may be provided to other components within the wearable system. By way of example, such data may be transmitted to components within non-head mounted unit 804 such as CPU 816 , including software modules for light field rendering controller 818 and alignment observer 820 .

Rendering controller 818 may use information from eye tracking module 814 to adjust the image displayed to the user by rendering engine 822 . For example, a rendering engine may represent a software module within GPU 830 that may provide images to display 208 . Rendering controller 818 may adjust the image displayed to the user based on the user's center of rotation or gaze center. In particular, the rendering controller 818 may use information about the user's eye-center to simulate a rendering camera (e.g., simulate collection of images from the user's eye-level), and direct the simulated rendering camera to Based on this, the image displayed to the user may be adjusted. Additional details regarding the operations that may be performed by the light field rendering controller 818 are provided in US patent application Ser. No. 16/250,931, which is incorporated herein by reference in its entirety.

A "rendering camera", sometimes referred to as a "pinhole eye camera" (or "eye-eye camera") or a "virtual pinhole camera" (or "virtual camera"), is possibly an object in the virtual world. database for use in rendering virtual image content. An object may have a location and orientation with respect to the user or wearer and potentially with respect to real objects in the environment surrounding the user or wearer. In other words, the rendering camera may represent a line of sight in the rendering space from which the user or wearer views 3D virtual content (eg, virtual objects) in the rendering space. A rendering camera may render a virtual image based on a database of virtual objects managed by a rendering engine and to be presented to the eye.

The virtual image may be rendered as if it were obtained from the user's or wearer's eye level. For example, a virtual image may have a specific set of intrinsic parameters (e.g. focal length, camera pixel size, principal point coordinates, distortion/distortion parameters, etc.) and a specific set of extrinsic parameters (e.g. translation components relative to the virtual world). and rotation components) as if captured by a pinhole camera (corresponding to a “rendering camera”).

A virtual image may be obtained from the line of sight of such a camera with the position and orientation of the rendering camera (eg, parametric parameters of the rendering camera). It follows that the system can define and/or adjust intrinsic and extrinsic rendering camera parameters. For example, the system can provide an image that appears to be from the user's or wearer's eye level, such that the virtual image appears as if captured from the eye level of a camera that has a specific location relative to the user's or wearer's eye level. A particular set of collateral rendering camera parameters may be defined so that it can be rendered.

The system may later dynamically adjust the collateral rendering camera parameters on-the-fly to maintain alignment with the specific location. Similarly, intrinsic rendering camera parameters may also be defined and dynamically adjusted over time. In some implementations, the image appears as if captured from the eye of a camera with an aperture (e.g., pinhole) at a specific location relative to the user's or wearer's eye (such as eye center or center of rotation or either location). rendered to.

In some embodiments, the system uses one rendering camera for the user's left eye as the user's eyes are physically separated from each other and thus consistently positioned at different locations. A separate rendering camera may be created or dynamically repositioned and/or reoriented for the user's right eye. In at least some implementations, virtual content rendered from the perspective of the rendering camera associated with the viewer's left eye is viewed by the user through the left eyepiece of a head-mounted display (e.g., head-mounted unit 802). may be presented to Virtual content rendered from the perspective of a rendering camera associated with the user's right eye may be presented to the user through the right eyepiece of such a head-mounted display.

Further details discussing the creation, adjustment, and use of rendering cameras in the rendering process are found in U.S. patent application Ser. , which is expressly incorporated herein by reference in its entirety).

In some examples, one or more modules (or components) of the system 800 (eg, light field rendering controller 818, rendering engine 822, etc.) can determine the position and orientation of the user's head and eyes (eg, (determined based on head pose and eye tracking data) may be used to determine the position and orientation of the rendering camera within the rendering space. That is, the system 800 effectively maps the position and orientation of the user's head and eyes to specific locations and angular positions within the 3D virtual environment, and directs the rendering camera to specific locations and angular positions within the 3D virtual environment. It can be placed and oriented to render virtual content for the user as would be captured by the rendering camera. Further details discussing the real-to-virtual world mapping process can be found in US patent application Ser. is expressly incorporated herein in its entirety by ).

As an example, rendering controller 818 may adjust the depth at which an image is displayed by selecting the depth plane or depth planes that are utilized at any given time to display the image. good. In some implementations, such depth plane switching may occur through adjustment of one or more intrinsic rendering camera parameters. For example, the light field rendering controller 818 may adjust the focal length of the rendering camera when performing depth plane switching or adjustment. As described in more detail below, depth planes may be switched based on the user's determined convergence-divergence movement or fixation depth.

Alignment observer 820 may use information from eye tracking module 814 to identify whether head-mounted unit 802 is properly positioned on the user's head. As an example, eye tracking module 814 may provide eye location information, such as the location of the center of rotation of the user's eyes, which indicates the three-dimensional position of the user's eyes relative to camera 252 and head-mounted unit 802 . The eye tracking module 814 uses the location information to determine whether the display 208 is properly aligned within the user's field of view, or whether the head-mounted unit 802 (or headset) has slipped or is in another position. It may be determined whether the user's eyes are similarly misaligned.

As an example, alignment observer 820 may be able to determine if head mounted unit 802 is slipping off the bridge of the user's nose. This may cause the display 208 to move away from and down the user's eyes (which may be undesirable). As another example, alignment observer 820 detects that head-mounted unit 802 has been moved above the bridge of the user's nose and thus display 208 has moved closer to and above the eyes from the user. can determine that As another example, alignment observer 820 may determine that head-mounted unit 802 is offset left or right with respect to the bridge of the user's nose. As another example, alignment observer 820 may determine that head mounted unit 802 is elevated above the bridge of the user's nose. As another example, alignment observer 820 may determine that head-mounted unit 802 has been moved in these or other ways away from a desired position or range of positions.

In general, alignment observer 820 may be able to determine whether head-mounted unit 802 in general, and display 208 in particular, are properly positioned in front of the user's eyes. In other words, alignment observer 820 determines whether the left display within display system 208 is properly aligned with the user's left eye, or whether the right display within display system 208 is properly aligned with the user's right eye. You can decide whether Alignment observer 820 determines whether head-mounted unit 802 is positioned and oriented within a desired range of positions and/or orientations relative to the user's eyes. may determine whether it is properly positioned.

In at least some embodiments, alignment observer 820 may generate user feedback in the form of alerts, messages, or other content. Such feedback is provided to the user, along with optional feedback on how to correct the misalignment (such as suggestions for adjusting the head-mounted unit 802 in a particular fashion), as well as the user's ability to adjust the head-mounted unit 802 . may signal any inconsistency in

Exemplary alignment observation and feedback techniques that may be utilized by alignment observer 820 are described in U.S. patent application Ser. document).

Example Eye Tracking Module A detailed block diagram of an exemplary eye tracking module 814a is shown in FIG. In some implementations, eye tracking module 814a may correspond to eye tracking module 814 of system 800 as described above with reference to FIG.

As shown in FIG. 9, eye tracking module 814a may include a variety of different sub-modules, may provide a variety of different outputs, and may provide a variety of different data available in tracking the user's eyes. may be used. As an example, the eye tracking module 814a may measure the geometry of the eye tracking camera 252 relative to the light source 848 and head-mounted unit 802, approximately 4 points between the center of the user's corneal curvature and the average center of rotation of the user's eye. Assumed eye dimensions 904, such as a typical distance of 0.7 mm or a typical distance between a user's center of rotation and eye-gaze center, and eye-tracking contingent, such as per-user calibration data 906, such as a particular user's interpupillary distance Available data may be used, including properties and intrinsic properties.

Additional examples of extrinsic properties, inherent properties, and other information that may be employed by eye tracking module 814a are described in U.S. patent application Ser. Docket No. MLEAP.023A7) (incorporated herein by reference in its entirety).

Image pre-processing module 910 may receive images from an eye camera, such as eye camera 252, and may perform one or more pre-processing (eg, adjustment) operations on the received images. As an example, image preprocessing module 910 may apply Gaussian blurring to the image. As another example, image preprocessing module 910 may downsample the image to a lower resolution. As another example, image preprocessing module 910 may apply a non-sharp mask. As another example, image preprocessing module 910 may apply an edge sharpening algorithm. As another example, image preprocessing module 910r may apply other suitable filters to aid in subsequent detection, localization, and labeling of flashes, pupils, or other features in images from eye camera 252. may apply. Image pre-processing module 910 may apply a low-pass filter, such as an open filter, or a morphological filter, which removes high frequency noise from pupillary boundaries 516a (see FIG. 5), etc., thereby Noise that can interfere with pupil and phosphene determination can be removed. Image preprocessing module 910 may output preprocessed images to pupil identification module 912 and flash detection and labeling module 914 .

Pupil identification module 912 may receive preprocessed images from image preprocessing module 910 and may identify regions of those images that contain the user's pupils. Pupil identification module 912 may, in some embodiments, determine the coordinates of the location of the user's pupil in the eye-tracking image from camera 252 or the coordinates of its center or centroid.

In at least some embodiments, the pupil identification module 912 identifies contours (e.g., pupil-iris boundary contours) in the eye-tracking image, identifies contour moments (e.g., center of mass), performs starburst pupil detection and /or apply a Canny edge detection algorithm to eliminate false correspondences, identify sub-pixel boundary points, and correct for eye camera distortion (e.g., distortion in images captured by eye camera 252) based on the intensity values. , applying a random sample shared term (RANSAC) iterative algorithm to fit an ellipse to the boundaries in the eye-tracked image, applying a tracking filter to the image, identifying the sub-pixel image coordinates of the centroid of the user's pupil, and good too.

Pupil identification module 912 may output pupil identification data to flash detection and labeling module 914, which may indicate areas of pre-processed image module 912 that have been identified as indicating the user's pupil. Pupil identification module 912 may provide the 2D coordinates of the user's pupil (eg, the 2D coordinates of the centroid of the user's pupil) in each eye-tracking image to flash detection module 914 . In at least some embodiments, pupil identification module 912 may also provide the same type of pupil identification data to coordinate system normalization module 918 .

Pupil detection techniques that may be utilized by the pupil identification module 912 are described in U.S. Patent Publication No. 2017/0053165, published February 23, 2017, U.S. Patent Publication No. 2017, published February 23, 2017. /0053166, and US Patent Application No. 15/693,975, published Mar. 7, 2019, each of which is hereby incorporated by reference in its entirety.

A flash detection and labeling module 914 may receive preprocessed images from module 910 and pupil identification data from module 912 . The flash detection module 914 uses this data to detect and/or detect flashes of light (eg, light from the light source 848 reflected from the user's eye) in regions of the pre-processed image that indicate the user's pupil. may be identified. As an example, the flash detection module 914 searches for bright regions in the eye-tracking image, sometimes referred to herein as "blobs" or local intensity maxima, within the vicinity of the user's pupil. You may

In at least some embodiments, flash detection module 914 may rescale (eg, enlarge) the pupil ellipse to include additional flashes. The flash detection module 914 may filter flashes by size and/or intensity. The flash detection module 914 may also determine the 2D location of each of the flashes within the eye-tracking image. In at least some embodiments, the flash detection module 914 may determine the 2D location of the flash relative to the user's pupil, which may also be referred to as the pupil-to-flash vector. Flash detection and labeling module 914 may label flashes and output pre-processed images with labeled flashes to 3D corneal center estimation module 916 . Flash detection and labeling module 914 may also communicate data such as pre-processed images from module 910 and pupil identification data from module 912 .

In some implementations, flash detection and labeling module 914 may also determine the light source (e.g., from among multiple light sources of the system, including infrared light sources 848a and 848b) that produced each identified flash of light. good. In these examples, flash detection and labeling module 914 may label the flash with information identifying the light source with which it is associated, and output a pre-processed image with the labeled flash to 3D corneal center estimation module 916. . In some implementations, the flash detection and labeling module 914 may be configured to utilize one or more of the flash detection techniques described above with reference to FIG. 7B.

Pupil and flash detection as performed by modules such as modules 912 and 914 may use any suitable technique. As an example, edge detection can be applied to eye images to identify glints and pupils. Edge detection can be applied by various edge detectors, edge detection algorithms, or filters. For example, a Canny edge detector can be applied to the image to detect edges such as lines in the image. The edge may include points located along the line corresponding to the maximum derivative. For example, pupil boundary 516a (see FIG. 5) can be located using a Canny edge detector.

Once the location of the pupil is determined, various image processing techniques can be used to detect the “pose” of the pupil 116 . The step of determining the eye pose of the eye image may also be referred to as the step of detecting the eye pose of the eye image. Posture may also be referred to as eye line-of-sight, facing direction, or orientation. For example, the pupil may be looking left toward the object, and the pupil pose may be classified as a left facing pose. Other methods can also be used to detect the location of the pupil or flash. For example, concentric rings can be located in the eye image using a Canny edge detector. As another example, an integral-differential operator can be used to find the limbal boundary of the pupil or iris. For example, the Daugman integral-differential operator, the Hough transform, or other iris segmentation techniques can be used to return a curve that estimates the boundaries of the pupil or iris.

A 3D corneal center estimation module 916 may receive pre-processed images from modules 910, 912, 914, including detected flash data and pupil identification data. A 3D corneal center estimation module 916 may use these data to estimate the 3D position of the user's cornea. In some embodiments, the 3D corneal center estimation module 916 calculates the corneal curvature or eye center of the user's corneal sphere, e.g., the center of an imaginary sphere that generally has a surface portion coextensive with the user's cornea. 3D positions can be estimated. 3D corneal center estimation module 916 provides data to coordinate system normalization module 918, optical axis determination module 922, and/or light field rendering controller 818 indicating the corneal sphere and/or estimated 3D coordinates of the user's cornea. You may

Techniques for estimating the location of ocular features such as the cornea or corneal sphere, which may be utilized by the 3D corneal center estimation module 916 and other modules within the wearable system of the present disclosure, filed Jan. 17, 2019 , US patent application Ser. No. 16/250,931, which is incorporated herein by reference in its entirety.

A coordinate system normalization module 918 may optionally be included within the eye tracking module 814a. Coordinate system normalization module 918 may receive data from 3D corneal center estimation module 916 indicating estimated 3D coordinates of the center of the user's cornea and/or the center of the user's corneal sphere, and may also receive data may be received from other modules. A coordinate system normalization module 918 may normalize the eye camera coordinate system, which can help compensate for slippage of the wearable device. For example, slippage may include slippage of the head-mounted component from its normal resting position on the user's head, which may be identified by alignment observer 820 .

A coordinate system normalization module 918 rotates the coordinate system so that the z-axis of the coordinate system (eg, the convergence-divergence motion depth axis) and the corneal center (eg, as indicated by the 3D corneal center estimation module 916). may be aligned. Module 918 may also translate the camera center (eg, the origin of the coordinate system) away from the corneal center by a predetermined distance. Exemplary distances may include 25mm, 30mm, 35mm, and the like. For example, module 918 may expand or contract the eye tracking image in response to determining whether eye camera 252 is closer or further than a predetermined distance. Using this normalization process, eye tracking module 814a is relatively independent of variations in the positioning of the headset on the user's head to establish consistent orientations and distances within the eye tracking data. may be possible.

Coordinate system normalization module 918 may also provide 3D coordinates of the center of the cornea (and/or corneal sphere), pupil identification data, and pre-processed eye tracking images to 3D pupil center locator module 920 . Further details of the operations that may be performed by the coordinate system normalization module 918 are provided in US patent application Ser. No. 16/250,931, which is incorporated herein by reference in its entirety.

The 3D pupil center locator module 920 stores data in a normalized or non-normalized coordinate system, including 3D coordinates of the center of the user's cornea (and/or corneal sphere), pupil location data, and pre-processed eye tracking images. may be received. A 3D pupil center locator module 920 may analyze such data to determine the 3D coordinates of the user's pupil center in a normalized or non-normalized eye camera coordinate system. The 3D pupil center locator module 920 stores the 2D location of the pupil centroid (eg, determined by module 912), the 3D location of the corneal center (eg, determined by module 916), the size of the corneal sphere of a typical user, and the corneal Based on assumed eye dimensions 904, such as the typical center-to-pupil center distance, and/or optical properties of the eye, such as the refractive index of the cornea (relative to the refractive index of air), or any combination thereof, the user may be determined in three dimensions. Further details of the operations that may be performed by the 3D pupil center locator module 920 are provided in US patent application Ser. No. 16/250,931, which is incorporated herein by reference in its entirety.

Optical axis determination module 922 may receive data from modules 916 and 920 indicating the 3D coordinates of the user's cornea and the user's pupil center. Based on such data, the optical axis determination module 922 may identify a vector from the location of the corneal center (e.g., from the center of the corneal sphere) to the user's pupil center, which is the distance of the user's eye. An optical axis can be defined. Optical axis determination module 922 may illustratively provide an output to modules 924, 928, 930, and 932 defining the user's optical axis.

A center of rotation (CoR) estimation module 924 stores data containing parameters of the optical axis of the user's eye (eg, data indicating the direction of the optical axis in a coordinate system with a known relationship to the head-mounted unit 802). may be received from module 922 . For example, CoR estimation module 924 may estimate the center of rotation of the user's eye. The center of rotation may indicate a point about which the user's eye rotates (eg, when the user's eye rotates left, right, up, and/or down). Although the eye may not rotate perfectly around a single point, assuming a single point may be sufficient in some embodiments. Additional points may be considered in some embodiments.

In at least some embodiments, the CoR estimation module 924 determines the distance between the center of the pupil (eg, identified by module 920) or the center of curvature of the cornea (eg, identified by module 916) (eg, identified by module 922). The center of rotation of the eye can be estimated by moving a certain distance along the optical axis towards the retina. This particular distance may be the assumed eye size 904 . As an example, the specified distance between the center of curvature of the cornea and the CoR may be approximately 4.7 mm. This distance may be varied for a particular user based on any relevant data, including the user's age, gender, visual prescription, other relevant characteristics, and the like. Additional discussion of the 4.7 mm value as an estimate for the distance between the center of curvature of the cornea and the CoR can be found in U.S. patent application Ser. incorporated).

In at least some embodiments, the CoR estimation module 924 may refine its estimate of the center of rotation of each of the user's eyes over time. As an example, as time progresses, the user may eventually rotate their eyes (e.g., to something else, closer, farther, or sometimes left, right, up, or down). viewing) will cause a shift in the respective optical axis of the eye. CoR estimation module 924 may then analyze the two (or more) optical axes identified by module 922 and locate the 3D point of intersection of those optical axes. The CoR estimation module 924 may then determine the center of rotation at the 3D point of that intersection. Such techniques may provide an estimate of the center of rotation with an accuracy that improves over time.

Various techniques may be employed to increase the accuracy of the CoR estimation module 924 and the determined CoR locations of the left and right eyes. As an example, the CoR estimation module 924 may estimate the CoR by finding the average point of the intersection of the optical axes determined over time for different eye poses. As another example, module 924 may filter or average the estimated CoR positions over time. As another example, module 924 may compute a moving average of estimated CoR positions over time. As another example, module 924 may apply a Kalman filter to the known dynamics of the eye and eye tracking system to estimate CoR position over time.

In some implementations, a least squares approach may be taken to determine one or more points of intersection of the optical axes. In such an implementation, the system may identify, at a given time, as the point of optical axis intersection, the location where the sum of the squared distances for a given set of optical axes is minimized. . As a specific example, module 924 calculates the determined CoR from an assumed CoR position (eg, 4.7 mm behind the center of the corneal curvature of the eye) of the user as eye tracking data is acquired for the user. The determined point of optical axis intersection and the hypothesized CoR position (corneal 4.7 mm from the center of curvature) may be calculated.

Under certain (e.g., substantially ideal) conditions, the 3D position of the true CoR of the user's eye with respect to the HMD will vary as the user moves the eye (e.g., the user's eye moves around its center of rotation rotation) should change over time by a negligible or minimal amount. For example, for a given set of eye movements, the 3D position of the true CoR of the user's eye (eg, relative to the HMD) is hypothetically as much as any other point along the optical axis of the user's eye over time. should not change. Thus, the farther a point along the point optical axis is from the true CoR of the user's eye, the more variation or variance its 3D position will exhibit over time as the user moves the eye. become. In some embodiments, the CoR estimation module 924 and/or other sub-modules of the eye tracking module 814a may utilize this statistical relationship to improve CoR estimation accuracy. In such embodiments, CoR estimation module 924 and/or other sub-modules of eye tracking module 814a identify variations in its CoR estimates that have low variability (e.g., low variance or standard deviation), That estimate of CoR3D position may be refined over time.

As a first example, a CoR estimation module 924 (eg, each associated with a user looking in a different direction) estimates the CoR based on the intersection of multiple different optical axes. The estimation module 924 introduces a common offset in each direction of the optical axis (e.g., displaces each axis by some uniform amount), and the intersection point where the offset optical axis has low variation (e.g., low dispersion or standard deviation). This statistical relationship (eg, the true CoR should have low variance) may be exploited by determining whether they intersect each other at . This can help correct for small systematic errors in calculating the orientation of the optical axis and refine the estimated position of the CoR to be closer to the true CoR.

A second example is that the CoR estimation module 924 calculates the CoR by moving a specified distance (eg, the distance between the center of curvature of the cornea and the CoR, etc.) along the optical axis or other axis. Presumably, it relates to an embodiment. With respect to this second example, the system is configured to reduce or minimize the variation, e.g., variance and/or standard deviation, of the estimated CoR position (e.g., eyes captured at different times). The specific distance between the center of curvature of the cornea and the CoR may be varied, optimized, adjusted, or otherwise adjusted over time.

For example, the CoR estimation module 924 first obtains a CoR position estimate using a particular distance value of 4.7 mm (eg, along the optical axis, from the center of curvature of the cornea), but If the true CoR of a given user's eye can be located 4.9 mm behind the center of the eye's corneal curvature (e.g., along the optical axis), the CoR position estimate obtained by the CoR estimation module 924 is The initial set may exhibit a relatively high amount of variability, eg variance or standard deviation. In response to detecting such a relatively high amount of variation (e.g., variance or standard deviation), CoR estimation module 924 has a lower amount of variation (e.g., variance or standard deviation) along the optical axis. , may determine one or more points. Accordingly, module 924 may identify the 4.9 mm distance as having the lowest variation (eg, variance or standard deviation) and may adjust the particular distance value utilized to 4.9 mm accordingly.

CoR estimation module 924, in response to detecting that the current CoR estimate has a relatively high amount of variation (eg, variance or standard deviation), calculates a lower variation (eg, variance and/or standard deviation ) may be determined. Module 924 may, of course, also determine alternative CoR estimates that have lower variability (eg, variance or standard deviation) after obtaining the initial CoR estimate. In some embodiments, such optimization/adjustment may occur gradually over time, while in other embodiments such optimization/adjustment may occur during an initial user calibration session. can be done. In embodiments where such a procedure is performed during a calibration procedure, the CoR estimation module 924 may not initially conform/stick to any assumed particular distance, but rather a set of eye-tracking data. Collected over time, statistical analysis is performed on a set of eye-tracking data to determine a particular distance value, and based on the statistical analysis, the smallest possible variability (e.g., global minimum) (e.g., (variance or standard deviation).

Additional discussion of the statistical relationships described above (e.g., true CoR should have low variance or standard deviation) and the significance of considering corneal refraction in determining pupillary position is No. 16/250,931, which is incorporated herein by reference in its entirety.

An interpupillary distance (IPD) estimation module 926 may receive data from the CoR estimation module 924 indicating the estimated 3D positions of the centers of rotation of the user's left and right eyes. The IPD estimation module 926 may then estimate the user's IPD by measuring the 3D distance between the centers of rotation of the user's left and right eyes. In general, the distance between the estimated CoR of the user's left eye and the estimated CoR of the user's right eye is approximately equal to the distance between the user's pupil centers when the user is looking at about optical infinity. could be. For example, this may refer to the optical axes of the user's eyes being substantially parallel to each other. This may refer to the typical definition of interpupillary distance (IPD).

A user's IPD may be used by various components and modules within the wearable system. As an example, the user's IPD is provided to alignment observer 820 to determine how well the wearable device is aligned with the user's eye (e.g., whether the left and right display lenses are properly spaced according to the user's IPD). may be used in assessing As another example, the user's IPD may be provided to the convergence-divergence motion depth estimation module 928 and used in determining the user's convergence-divergence motion depth. Module 926 may employ various techniques, such as those discussed in connection with CoR estimation module 924, to increase the accuracy of the estimated IPD. As an example, the IPD estimation module 924 applies filtering, averaging over time, weighted averaging including assumed IPD distance, Kalman filter, etc. as part of estimating the user's IPD in an accurate manner. may

Convergence-divergence motion depth estimation module 928 may receive data from various modules and sub-modules within eye tracking module 814a (as shown in connection with FIG. 9). In particular, the convergence-divergence motion depth estimation module 928 generates an estimated 3D position of the pupil center (eg, as provided by module 920 described above), an estimated 3D position (eg, by module 922 described above) One or more determined parameters of the optical axis (as provided by module 924 described above), an estimated 3D position of the center of rotation (eg, as provided by module 924 described above), Estimated IPD (eg, Euclidean distance between estimated 3D positions of centers of rotation), and/or (eg, module 922 and/or module 930 described below), as provided by module 926 described below. Data indicative of one or more determined parameters of optics and/or visual axis may be employed, such as provided by .

The convergence-divergence motion depth estimation module 928 may detect or otherwise obtain a measurement of the user's convergence-divergence motion depth. Convergence-divergence depth of motion may indicate the distance from the user at which the user's eyes are focused. As an example, when a user is looking at an object 3 feet in front of them, the user's left and right eyes have a convergence-divergence depth of motion of 3 feet. As another example, when the user is looking at a distant scene, the user's left and right eyes have infinite convergence-divergence depth of motion. In this example, the optical axes of the user's eyes may be substantially parallel to each other such that the distance between the centers of the user's pupils may be substantially equal to the distance between the centers of rotation of the user's left and right eyes.

In some implementations, the convergence-divergence motion depth estimation module 928 utilizes data indicating an estimated center of the user's pupil (eg, as provided by module 920) to estimate the user's pupil may determine the 3D distance between the centers to be measured. By comparing the estimated IPD (e.g., the Euclidean distance between the estimated 3D positions of the centers of rotation) to the 3D distances between such determined pupil centers, the convergence-divergence motion depth estimation module 928 Measurements of convergence-divergence depth may be obtained.

In addition to the 3D distance between pupil centers and the estimated IPD, the convergence-divergence motion depth estimation module 928 utilizes known, assumed, estimated and/or determined geometries to Convergence-divergence motion depth may be calculated. As an example, module 928 may combine the 3D distance between pupil centers, the estimated IPD, and the 3D CoR position in a triangulation calculation to estimate (eg, determine) the user's convergence-divergence depth of motion. . In some embodiments, an estimate of such determined 3D distance between pupil centers relative to estimated IPD serves to indicate the user's current convergence-divergence depth of motion relative to optical infinity. obtain. In some examples, the convergence-divergence motion depth estimation module 928 uses the estimated distance between the estimated centers of the user's pupils for the purpose of obtaining such a measure of the convergence-divergence motion depth. data may be received or accessed indicating a 3D distance to the object.

In some embodiments, the convergence-divergence motion depth estimation module 928 may estimate the convergence-divergence motion depth by comparing the user's left and right optical axes. In particular, the convergence-divergence motion depth estimation module 928 calculates the distance from the user at which the user's left and right optical axes intersect (or the projections of the user's left and right optical axes on a plane such as a horizontal plane intersect). The convergence-divergence motion depth may be estimated by locating . Module 928 may utilize the user's IPD in this calculation by setting zero depth to the depth at which the user's left and right optical axes are separated by the user's IPD. In at least some embodiments, the convergence-divergence motion depth estimation module 928 may determine the convergence-divergence motion depth by triangulating eye tracking data along with known or derived spatial relationships. good.

In some embodiments, the convergence-divergence motion depth estimation module 928 estimates the user's convergence-divergence motion depth based on the intersection of the user's visual axis, as opposed to the optical axis as described above. can be estimated. This may provide a more accurate indication of the distance over which the user is focusing.

In at least some embodiments, the eye tracking module 814 a may include an optical axis to visual axis mapping module 930 . As discussed in more detail in connection with FIG. 10, the user's optics and visual axis are generally not aligned. The visual axis is the axis along which a person is looking, while the optical axis is defined by the person's lens and pupil center and may pass through the center of the person's retina. In particular, a user's visual axis is generally defined by the location of the user's fovea, which may be offset from the center of the user's retina, thereby resulting in different optics and visual axes. In at least some of these embodiments, the eye tracking module 814 a may include an optical axis to visual axis mapping module 930 . The optical and visual axis mapping module 930 corrects for differences between the user's optical and visual axes and transfers information about the user's visual axis to the convergence and divergence motion depth estimation module 928 and the light field rendering controller 818. It may be provided to other components in the wearable system such as.

In some embodiments, module 930 includes a typical offset of about 5.2° inward (nasal, i.e., toward the user's nose) between the optical axis and the visual axis. eye size 904 may be used. In other words, module 930 rotates the user's left optical axis 5.2° (nasally) right toward the nose and the user's right optical axis to There may be a 5.2° (nasal) left deviation toward the nose. In other examples, module 930 may utilize per-user calibration data 906 in mapping the optical axis (eg, as illustrated by module 922 described above) to the visual axis. As an additional example, module 930 may steer the user's optical axis nasally between 4.0° and 6.5°, between 4.5° and 6.0°, between 5.0° and 5.4°, etc., or Any range formed by any of these values may be shifted. In some arrangements, module 930 may apply shifts based, at least in part, on characteristics of a particular user, such as its age, gender, visual prescription, or other relevant characteristics, and/or Or, the shift may be applied, at least in part, based on a calibration process for a particular user (eg, to determine a particular user's optical axis-visual axis offset). In at least some embodiments, module 930 also shifts the origin of the left and right optical axes to correspond to the user's CoP (as determined by module 932) instead of the user's CoR. good.

An optional center-of-sight (CoP) estimation module 932, when provided, may estimate the location of the user's left and right center-of-sight (CoP). The CoP can be a useful location for wearable systems, and in at least some embodiments is a location directly in front of the pupil. In at least some embodiments, the CoP estimation module 932, based on the 3D location of the user's pupil center, the 3D location of the user's corneal curvature center, or such suitable data, or any combination thereof: The locations of the user's left and right eye gaze centers may be estimated. As an example, the user's CoP may be approximately 5.01 mm in front of the center of corneal curvature and approximately 2.97 mm behind the outer surface of the user's cornea along the optical or visual axis. 5.01 mm may represent the distance from the corneal spherical center in a direction toward the cornea of the eye and along the optical axis. The user's gaze center may be directly in front of the pupil center. As an example, the user's CoP is less than about 2.0 mm from the user's pupil, less than about 1.0 mm from the user's pupil, or less than about 0.5 mm from the user's pupil, or any of these values. It can be in any range. As another example, the gaze center may correspond to a location within the anterior chamber of the eye. As other examples, the CoP is between 1.0 mm and 2.0 mm, about 1.0 mm, between 0.25 mm and 1.0 mm, between 0.5 mm and 1.0 mm, or between 0.25 mm and 0.5 mm. good.

The line-of-sight center described herein (potentially as a desired location for the rendering camera pinhole and anatomical location within the user's eye) reduces and/or eliminates undesirable parallax shifts can be a position that serves to In particular, the optics of the user's eye can be roughly compared to a theoretical system formed by a pinhole in front of a lens that projects onto the screen, where the pinhole, lens, and screen are each the pupil of the user. / Corresponds approximately to the iris, lens, and retina. In addition, two point sources (or objects) at different distances from the user's eye are rigidly rotated about the pinhole aperture (e.g., to radii of curvature equal to their respective distances from the pinhole aperture). ), it may be desirable that there is little or no parallax shift.

Therefore, it would follow that the CoP should be located at the pupil center of the eye (and such a CoP may be used in some embodiments). However, the human eye contains, in addition to the lens and pinhole of the pupil, the cornea, which provides additional refractive power for light to propagate towards the retina. The anatomical equivalent of a pinhole within the theoretical system described in this paragraph is therefore a pinhole in a user's eye, located between the outer surface of the cornea of the user's eye and the center of the pupil or iris of the user's eye. can be the region of For example, the anatomical equivalent of a pinhole may correspond to a region within the anterior chamber of the user's eye. For various reasons discussed herein, it may be desirable to set the CoP at such a location within the anterior chamber of the user's eye. The derivation and significance of CoP is described in US patent application Ser. No. 16/250,931, which is hereby incorporated by reference in its entirety.

As discussed above, the eye tracking module 814a calculates the estimated 3D positions of the left and right eye centers of rotation (CoR), the depth of convergence and divergence motion, the left and right eye optical axes, the 3D positions of the user's eyes. , the 3D positions of the user's left and right centers of corneal curvature, the 3D positions of the user's left and right pupil centers, the 3D positions of the user's left and right gaze centers, the user's IPD, etc. It may be provided to other components such as the rendering controller 818 and alignment observer 820 . Eye tracking module 814a may also include other sub-modules that detect and generate data associated with other aspects of the user's eye. As an example, the eye tracking module 814a includes a blink detection module that provides a flag or other alert each time the user blinks, and a blink detection module that detects when the user's eye saccades (e.g., quickly shifts focus to another point). and a saccade detection module, which provides a flag or other alert each time it deviates.

FIG. 10 illustrates a block diagram of an exemplary eye tracking module 814b. In some implementations, eye tracking module 814b may correspond to eye tracking module 814 of system 800, as described above with reference to FIG. As shown in FIG. 10, the eye tracking module 814b may include a variety of different sub-modules, may provide a variety of different outputs, and may use a variety of available data when tracking the user's eyes. may be used. More specifically, eye tracking module 814b includes deep segmentation network 1007 (eg, one or more machine learning models), contour determination module 1009, centroid determination module 1011, pupil identification module 1012, and flash detection. and labeling module 1014; 3D corneal center estimation module 1016; 3D pupil center locator module 1020; optical axis determination module 1022; CoR estimation module 1024; and a diffuse motion depth estimation module 1032 . In some implementations, at least some of the operations performed by one or more of sub-modules 1012, 1014, 1016, 1020, 1022, 1024, 1030, and 1032 in FIG. At least some of the operations performed by one or more of the sub-modules 912, 914, 916, 920, 922, 924, 930, and 932 as described above with reference to FIG. can correspond to In some examples, eye tracking module 814b is configured to perform one or more additional sub-modules configured to perform one or more of the operations described above with reference to FIG. may include

In some implementations, deep segmentation network 1007 is trained or otherwise configured to perform one or more of the eye image segmentation operations described above with reference to FIGS. 1A-7B. may correspond to a neural network or other model that Exemplary neural networks may include fully connected neural networks, convolutional neural networks, and the like. Thus, deep segmentation network 1007 may receive an image of the user's eye from eye camera 252 as an input and may provide eye image segmentation data as output to contour determination module 1009 . Contour determination module 1009 may, for example, identify contours or boundaries of the sclera, iris, and/or pupil of the user's eye based on the eye image segmentation data generated by deep segmentation network 1007 . This information may also be referred to herein as iris segmentation data and/or pupil segmentation data. In some implementations, contour determination module 1009 may provide data to flash detection and labeling module 1014 indicating the determined contour of the user's iris. Additionally, the contour determination module 1009 may output data to the centroid determination module 1011 indicating the determined contour of the user's pupil in some embodiments.

Data indicative of the determined outline of the iris is provided by the contour determination module 1009 as an input to the flash detection and labeling module 1014. In implementations, the flash detection and labeling module 1014 uses Such data may then be used to narrow the search space with respect to flashes. More specifically, in some embodiments, flash detection and labeling module 1014 identifies a particular eye image received from eye camera 252 based on data received from contour determination module 1009 . Regions may be identified and the presence of flashes within the identified specific regions may be searched. Such identified specific regions may correspond, at least in part, to regions of the iris, for example. In some implementations, the phosphene detection and labeling module 1014 detects phosphenes such that other regions of a given eye image are effectively excluded from phosphenes search and/or discarded entirely. The search can be limited to such areas. Accordingly, the flash detection and labeling module 1014 can perform the detection in a faster and/or less computationally intensive manner by leveraging data indicative of the determined contour of the iris as provided by the contour determination module 1009. In , it may be possible to detect and label flashes of light.

As will be described in further detail below with reference to FIG. 11, in some embodiments, the flash detection and labeling module 1014 determines the contour determination module 1009 based on one eye image. Data indicating the determined contour of the iris may be utilized to identify specific regions within one or more subsequently received eye images. In other words, in some embodiments, flash detection and labeling module 1014 determines the iris in the (n-1)th eye image captured by eye camera 252 as provided by contour determination module 1009. Specific regions within the n th eye image captured by the eye camera 252 may be identified based on the data showing the contours drawn. Given that eye image segmentation can be relatively computationally and temporally intensive, the operations of sub-modules 1007, 1009, and 1011 are thus separated from other sub-modules of eye tracking module 814b. Decoupling can serve to reduce overall system latency.

As noted above, in some embodiments, contour determination module 1009 may output data indicative of the determined contour of the user's pupil to centroid determination module 1011 . The centroid determination module 1011 may then use such data to determine the contour instant, the center of mass, or the centroid of the pupil. In some implementations, the centroid determination module 1011 may provide data indicating the centroid of the user's pupil to the pupil identification module 1012 . In such implementations, pupil identification module 1012 may, in some embodiments, utilize such data to provide a starting point for identifying pupils within a given eye image. For example, pupil identification module 1012 identifies specific locations within a given eye image received from eye camera 252 based on data received from contour determination module 1009 and identifies specific locations for the presence of pupil boundaries. It may also search outward from the starting location (eg, using the starburst algorithm).

In embodiments where a starburst algorithm is employed, the pupil identification module 1012 may use the identified location described above as the location from which the starburst begins (eg, the origin of the burst). good. Also, as described in further detail below with reference to FIG. 11, in some embodiments, pupil identification module 1012 also includes pupil Data indicative of the determined centroid of the may be utilized to identify specific locations within one or more subsequently received eye images. In other words, in some embodiments, pupil identification module 1012 uses the determined centroid of the pupil in the (n-1)th eye image captured by eye camera 252 as provided by centroid determination module 1011. A specific location within the eye in the n-th image captured by the eye camera 252 may be identified based on data indicative of . In such an embodiment, pupil identification module 1012 subsequently searches outward from the identified specific location within the nth eye image for the presence/location of a pupil boundary within the nth eye image. (eg, using the starburst algorithm). As noted above, latency savings can thus be achieved by decoupling the operations of sub-modules 1007, 1009, and 1011 from other sub-modules of eye tracking module 814b.

FIG. 11 is a flow chart illustrating an exemplary process 1100 for performing eye tracking with reduced latency. Process 1100 can be implemented by embodiments of the wearable display system described herein, for example, using eye tracking module 814b described above with reference to FIG. In various implementations of process 1100, the blocks described below may be performed in any suitable order or sequence, blocks may be combined or rearranged, or other blocks may be can be added.

In some implementations, process 1100 produces one or more cameras configured to capture an image of a user's eye and flashes of light within the image of the user's eye captured by the one or more cameras. a head comprising a plurality of light sources (e.g., infrared light sources) configured to illuminate the eyes of a user and one or more processors coupled to one or more cameras in a manner as to It may also be implemented by an on-board system. In at least some such implementations, some or all of the operations of process 1100 may be performed, at least in part, by one or more processors of the system.

At blocks 1102-1106 (eg, blocks 1102, 1104, and 1106), process 1100 may begin with acquiring a first eye image captured by an eye-tracking camera. This may correspond, as an example, to one or more sub-modules of eye tracking module 814b that acquire an image of the user's eye from eye tracking camera 252 at the start of a new eye tracking session. Such a new eye-tracking session may be initiated, for example, in response to the HMD turning on or launching a particular application.

At block 1108 , process 1100 includes determining whether the eye image acquired at block 1106 represents the first eye image captured and acquired subsequent to initiation of process 1100 at block 1102 . Given that, in this instance, the operations of block 1108 are performed immediately after those of blocks 1102-1106, and process 1100 is such that, in this instance, the eye image acquired at block 1106 is the Determining to represent the first eye image captured and acquired subsequent to initiation may also be included. Accordingly, process 1100 may proceed to blocks 1110 and 1120 in this instance. In some implementations, process 1100 may include proceeding to blocks 1110 and 1120 at the same time.

At block 1110, process 1100 may include segmenting the first eye image using a deep segmentation network. This involves segmenting the eye image using, for example, one or more of the techniques described above with reference to FIGS. 1A-7B, as described above with reference to FIG. deep segmentation network 1007 of eye tracking module 814b.

At block 1112, process 1100 may include determining the contours of the segmented iris and pupil in the first eye image. This is the eye tracking module 814b as described above with reference to FIG. It may correspond to contour determination module 1009 .

At block 1114, process 1100 may include determining a centroid of the segmented pupil in the first eye image based on the determined contour of the segmented pupil. This, as an example, determines the contour moments, center of mass, or center of mass of the segmented pupil in the eye image based on data provided by the contour determination module 1009, described above with reference to FIG. It may correspond to the centroid determination module 1011 of the eye tracking module 814b as described.

At blocks 1120 and 1108, process 1100 may include acquiring a second eye image captured by the eye tracking camera. As noted above, in some implementations the acts associated with blocks 1120 and 1108 may be performed concurrently with one or more of the acts associated with blocks 1110-1114. At block 1108 , process 1100 includes determining whether the eye image acquired at block 1106 represents the first eye image captured and acquired following initiation of process 1100 at block 1102 . Given that, in this instance, the operations of block 1108 may be performed subsequent to (e.g., immediately after) those of blocks 1120 and 1106, and process 1100 is, in this instance, the latest Determining that the eye image does not represent the first eye image captured and acquired subsequent to initiation of process 1100 at block 1102 may also include determining. Accordingly, process 1100 may proceed to blocks 1109 and 1110 in this instance. In some implementations, process 1100 may include proceeding to blocks 1109 and 1110 at the same time.

At block 1109, process 1100 may include identifying a region of interest in the second eye image based on the determined contour of the segmented iris in the first eye image. This may correspond to flash detection and labeling module 1014 of eye tracking module 814b, as an example. As explained above with reference to FIG. 10, this uses the data generated by the contour determination module 1009, based on the segmentation of the previous eye image, to determine the given A step of identifying regions of the eye image may be included. Given that relatively small eye movements are likely to occur between successive eye images, the iris contour determined based on the segmentation of the first eye image is the second can be sufficiently useful for iris boundaries in eye images.

At block 1111, process 1100 may include searching the identified region of interest in the second image for flashes. For example, one or more locations (eg, a set of locations) within the second image may be detected as representing flashes of light. As explained above, bright regions in the second image may be identified (eg, blobs or local intensity maxima). In some embodiments, flash locations may be identified as local intensity maxima within two image dimensions (eg, X and Y positions in the second image). In these embodiments, by way of example, a Gaussian distribution may be determined for the flashlight via determining the maximum intensity value within a bounding box surrounding the estimated flashlight. Image locations that correspond to maximum intensity values may be assigned as flash locations. An estimated flash of light may be determined based on an increase in image intensity (eg, brightness) compared to another portion of the second image.

The operations associated with block 1111 may also correspond to those performed by flash detection and labeling module 1014 of eye tracking module 814b as described above with reference to FIG.

At block 1113, process 1100 may include estimating the location of the 3D corneal center in the second eye image based on the detected flashes within the identified region of interest. This may correspond to the 3D corneal center module 1016 of the eye tracking module 814b as an example. As described above with reference to FIG. 10, this involves estimating the location of the 3D corneal center within a given eye image based on data provided by the flash detection and labeling module 1014. may contain.

At block 1115, process 1100 may include identifying the pupil in the second image based on the determined centroid of the segmented pupil in the first eye image. This may correspond to pupil identification module 1012 of eye tracking module 814b, as an example. As described above with reference to FIG. 10, this uses the data generated by the centroid determination module 1011 based on the segmentation of the previous eye image and its determined contours to obtain a given identifying a pupil in the eye image of the . Given that relatively small eye movements are likely to occur between consecutive eye images, the pupil centroid determined based on the segmentation of the first eye image is the first for pupil identification purposes. It can be quite informative for the centroid of the pupil in two eye images. In fact, as long as the pupil centroid determined based on the segmentation of the first eye image falls on some location within the region of the pupil in the second eye image, then An outer boundary can be identified within sufficiently high accuracy (eg, using a starburst algorithm). In some implementations, the acts associated with block 1115 may be performed in parallel with the acts associated with blocks 1109-1113. In some embodiments, one or more of the operations associated with block 1115 may be performed prior to one or more of the operations associated with blocks 1109-1113.

At block 1117, process 1100 calculates the 3D pupil in the second eye image based on the estimated location of the 3D corneal center in the second eye image and the pupil as identified in the first eye image. A step of estimating the location of the center may be included. This may correspond to 3D pupil center locator module 1020 of eye tracking module 814b, as an example. As described above with reference to FIG. 10, this uses data provided by the 3D central corneal module 1016 and data provided by the pupil identification module 1012 to generate a 3D image within a given eye image. A step of estimating the location of the corneal center may be included.

At block 1119, process 1100 may include estimating the position and orientation of the eye optical axis in the second eye image based on the estimated 3D corneal and pupil center locations in the second eye image. . This may correspond to optical axis determination module 1022 of eye tracking module 814b, as an example. As described above with reference to FIG. 10, this determines the determining the position and orientation of the optical axis of the eye.

Process 1100 may then proceed to blocks 1120 and 1106 to acquire the next eye image (eg, the third eye image) captured by the eye tracking camera for subsequent processing. In some implementations, process 1100 may then proceed to blocks 1120 and 1106 prior to completing one or more of the operations associated with blocks 1117 and/or 1119 .

Performance of process 1100 may continue for the duration of the eye tracking session in some examples. Thus, from the second eye image onwards, the operations associated with blocks 1109, 1111, 1113, 1115, 1117, and 1119 are the previously acquired eye images (eg, segments of the (n−1)th eye image). may be applied to the most recently acquired eye image (e.g., the nth eye image) using the data generated based on the transformation.Thus, a system performing the operations of process 1100 may: There is no need to wait for the segmentation operation to be completed before acquiring and processing the next eye image, which can effectively serve to reduce latency.In some implementations, the block The operations associated with 1109, 1111, 1113, 1115, 1117, and 1119 use the most recent segmentation data available (e.g., data generated by one or more of sub-modules 1007, 1009, and 1011). ) may be used, so if additional and/or more powerful computing resources are available for a system configured to perform the operations of process 1100, blocks 1109, 1111, 1113, 1115, Operations associated with 1117 and 1119 may be applied to the nth eye image using segmentation data generated based on the nth eye image.

Similarly, when a relatively large computational load is imposed on a system configured to perform the operations of process 1100, the operations associated with blocks 1109, 1111, 1113, 1115, 1117, and 1119 are , for example, data generated based on the (n-1)th eye image, (n-2)th eye image, (n-3)th eye image, etc. The segmentation data (eg, the most recent segmentation data) may be used and applied to the nth eye image. Thus, data generated based on the second or first image may be used for the third image.

In some implementations, process 1100 includes one or more sub-modules corresponding to the operation of one or more other sub-modules of eye tracking modules 814a and 814b as described above with reference to FIGS. Additional actions may be included.

In at least some implementations where the operations associated with blocks 1109, 1111, 1113, 1115, 1117, and 1119 use the most recent segmentation data available, the operations associated with blocks 1110, 1112, and 1114 are , is not necessarily performed for each eye image acquired. For example, the operations associated with blocks 1110, 1112, and 1114 may be performed on every other eye image acquired. However, in such embodiments, the operations associated with blocks 1109, 1111, 1113, 1115, 1117, and 1119 may be performed for each eye image acquired. Thus, the operations associated with blocks 1110, 1112, and 1114 may, for example, be performed at a first frequency, while the operations associated with blocks 1109, 1111, 1113, 1115, 1117, and 1119 are It may be performed at a second frequency that is a multiple of the first frequency (eg, twice that of the first frequency). For example, in some embodiments, the operations associated with blocks 1109, 1111, 1113, 1115, 1117, and 1119 may be performed for each eye image acquired at a rate of 60 Hz, while blocks 1110, 1112 , and 1114 may be performed on every other eye image acquired at a rate of 30 Hz. In some embodiments, the operations associated with blocks 1110, 1112, and 1114 are every other eye image acquired, every third eye image acquired, every fourth eye image acquired, , or some other interval. Other configurations are also possible.

The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of the present disclosure, some non-limiting features will now be briefly discussed. The following paragraphs describe various exemplary implementations of the devices, systems, and methods described herein. A system of one or more computers is configured to perform a particular operation or action by having software, firmware, hardware, or a combination thereof installed on the system that, when run, causes the system to perform an action. can be One or more computer programs can be configured to perform a particular operation or action by containing instructions which, when executed by a data processing device, cause the device to perform the action. Embodiments described herein include at least the following examples below.

Example 1: A head-mounted system, a camera configured to capture an image of a user's eye and illuminating the user's eye such that a flash of light is represented in the image of the user's eye and a first image of a user's eye from a camera and trained to generate iris segmentation data and pupil segmentation data given the image of the eye. A machine learning model is provided with a first image as input, following the first image, a second image of the user's eye is obtained from the camera, and based on the iris segmentation data, one detecting a set of one or more locations in the second image where the flashes of light are represented and, based on the pupil segmentation data, identifying regions of the second image where the pupil of the user's eye is represented; , configured to determine an eye pose of the user based, at least in part, on the detected set of one or more flash locations in the second image and the identified region of the second image. and one or more processors.

Example 2: The one or more processors further obtain iris segmentation data and pupil segmentation data for the second image and obtain a third image of the user's eye from the camera via the machine learning model. , respectively, detect a set of one or more locations where one or more flashes are indicated in the third image based on iris segmentation data generated by the machine learning model for the second image; identifying a region of the third image in which the pupil of the user's eye is represented based on the pupil segmentation data generated by the machine learning model for the image of the image; 2. The system of example 1, configured to determine a user's eye pose based on the above detected set of flash locations and the identified region of the third image.

Example 3: The one or more processors are further configured to identify a second region of the second image based on the iris segmentation data, wherein the one or more flashes in the second image are indicated. The one or more processors search a second region of the second image for distinct locations exhibiting maximal intensity values in the second region to detect a set of one or more locations that are subject to The system of example 1, wherein:

Example 4: The one or more processors are further configured to avoid searching regions of the second image outside the specified region for the set of one or more locations where the one or more flashes of light are shown. , the system of Example 3.

Example 5: The one or more processors are further configured to identify an iris contour of the user's eye in the first image based on the iris segmentation data; To identify different regions of the two images, the one or more processors identify a third region of the second image based on the identified contours of the iris of the user's eye in the first image. 4. The system of example 3, wherein the system is configured to:

Example 6: The one or more processors are further configured to identify a centroid of the pupil of the user's eye in the first image based on the pupil segmentation data, and identify a region of the second image To do so, the one or more processors identify a second region of the second image representing the pupil of the user's eye based on the identified centroid of the pupil of the user's eye in the first image. Example 1. The system of example 1, wherein the system is configured to:

Example 7: To identify a second region of a second image, the one or more processors generate a second image based on the identified centroid of the pupil of the user's eye in the first image. 7. The system of example 6, configured to identify a location in the second image and retrieve the pupillary boundary from the location identified in the second image.

Example 8: To retrieve a pupil boundary from a location identified in a second image, the one or more processors, based on the location identified in the second image assigned as a starting point: Example 7. The system of example 7, configured to perform a starburst pupil detection process.

Example 9: Determine the user's eye pose based, at least in part, on the detected set of one or more flash locations in the second image and the identified region of the second image To that end, the one or more processors determine the user's 2. The system of example 1, configured to obtain a position and orientation estimate of the optical axis of the eye.

Example 10: The one or more processors further estimate a three-dimensional location of the cornea of the user's eye in the second image based on the detected set of one or more flash locations in the second image. The one or more processors configured to obtain values to determine the pose of the user's eye, at least in part, an estimated location of the cornea of the user's eye in the second image; 2. The system of example 1, configured to determine a pose based on the identified region of the second image.

Example 11: The one or more processors are further configured, based on the estimated location of the cornea of the user's eye in the second image, The one or more processors are configured to obtain an estimate of the three-dimensional location of a pupil of an eye of a user, and to determine a pose of the user's eye, the one or more processors determine, at least in part, a second 11. The system of example 10, wherein the system is configured to determine an estimated location of a cornea of a user's eye in the image of and an estimated location of a pupil of the user's eye in the second image.

Example 12: The one or more processors further acquire a third image of the user's eye, each representing one or more flashes based on the iris segmentation data most recently generated by the machine learning model. detecting a set of one or more locations in the third image where the pupil of the user's eye is shown based on the pupil segmentation data most recently generated by the machine learning model. and a second pose of the user's eye based, at least in part, on the detected set of one or more flash locations in the third image and the identified region of the third image Example 1. The system of example 1, configured to determine .

Example 13: The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprises iris segmentation data and pupil segmentation data generated by the machine learning model for the second image. 13. The system according to 12.

Example 14: The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprises iris segmentation data and pupil segmentation data generated by the machine learning model for the first image. 13. The system according to 12.

Example 15: The one or more processors are further configured to provide the second image as an input to the machine learning model, the iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprising: 13. The system of example 12, comprising iris segmentation data and pupil segmentation data generated by the machine learning model for the third image.

Example 16: The system of Example 12, wherein the one or more processors are further configured to avoid providing the second image as an input to the machine learning model.

Example 17: A method implemented by a head-mounted system of one or more processors, wherein the head-mounted system illuminates the eye of a user such that a flash of light is represented in the image of the eye a machine learning model configured to obtain a first image of a user's eye and trained to generate iris segmentation data and pupil segmentation data given the image of the eye; providing as input one image; obtaining a second image of the user's eye; and based on the iris segmentation data, one in the second image in which one or more flashes of light are represented. detecting the above set of locations; identifying, based on the pupil segmentation data, a region of the second image in which the pupil of the user's eye is represented; and determining an eye pose of the user based on the detected set of one or more flash locations of and the identified regions of the second image.

Example 18: Obtaining, via a machine learning model, iris segmentation data and pupil segmentation data for a second image; obtaining a third image of the user's eye; detecting a set of one or more locations where one or more flashes are exhibited in the third image based on iris segmentation data generated by the machine learning model for the image; identifying, at least in part, a region of a third image in which the pupil of the user's eye is represented, based on the pupil segmentation data generated by the learning model; 18. The method of example 17, further comprising determining an eye pose of the user based on the detected set of flash locations and the identified regions of the third image.

Example 19: Further comprising identifying a second region of the second image based on the iris segmentation data, the set of one or more locations in the second image where the one or more flashes are shown 18. The method of example 17, wherein the method further comprises searching the second region of the second image for distinct locations exhibiting maximum intensity values within the second region, to detect .

Example 20: The head-mounted system is configured to avoid searching regions of the second image outside a specified region for a set of one or more locations where one or more flashes of light are shown. The method described in Example 19.

Example 21: Further comprising identifying iris contours of the user's eye in the first image based on the iris segmentation data, and identifying different regions of the second image based on the iris segmentation data. 20. As in example 19, the method further comprises identifying a third region of the second image based on the identified contour of the iris of the user's eye in the first image, to the method of.

Example 22: Further comprising the step of identifying a centroid of the pupil of the user's eye in the first image based on the pupil segmentation data, and to identify a region of the second image, the method further comprising: 18. As described in example 17, comprising identifying a second region of the second image in which the pupil of the user's eye is represented based on the identified centroid of the pupil of the user's eye in the first image. the method of.

Example 23: To identify a second region of a second image, the method includes, based on the identified centroid of the pupil of the user's eye in the first image, the location in the second image: 23. The method of example 22, comprising identifying , and retrieving the pupillary boundary from the identified locations in the second image.

Example 24: To retrieve the pupillary boundary from the location identified in the second image, the method uses the location identified in the second image to be assigned as the starting point, the starburst pupil 24. The method of example 23, comprising performing a detection process.

Example 25: Determining a user's eye pose based, at least in part, on a detected set of one or more flash locations in a second image and an identified region of the second image To this end, the method comprises, at least in part, based on a detected set of one or more flash locations in the second image and an identified region of the second image, an optical image of the user's eye. 18. The method of example 17, comprising obtaining axis position and orientation estimates.

Example 26: The method further obtains an estimate of the three-dimensional location of the cornea of the user's eye in the second image based on the detected set of one or more flash locations in the second image. and to determine the pose of the user's eye, the method includes, at least in part, an estimated location of the cornea of the user's eye in the second image and an identified position of the second image. 18. The method of example 17, comprising determining a pose based on the region and the region.

Example 27: The method further includes determining the location of the user's eye in the second image based on the estimated location of the cornea of the user's eye in the second image and the identified region of the second image. the method includes obtaining an estimate of the three-dimensional location of the pupil of the user's eye in the second image to determine the pose of the user's eye. 27. The method of example 26, comprising determining a pose based on the location of the eye and an estimated location of the pupil of the user's eye in the second image.

Example 28: The method further comprises acquiring a third image of the user's eye, each representing one or more flashes based on iris segmentation data most recently generated by the machine learning model. detecting a set of one or more locations in a third image where the pupil of the user's eye is shown based on pupil segmentation data most recently generated by the machine learning model; identifying a region; and based, at least in part, on the detected set of one or more flash locations in the third image and the identified region of the third image. 18. The method of example 17, comprising the step of determining 2 poses.

Example 29: The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprises iris segmentation data and pupil segmentation data generated by the machine learning model for the second image. 28. The method according to 28.

Example 30: The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprises iris segmentation data and pupil segmentation data generated by the machine learning model for the first image. 28. The method according to 28.

Example 31: The method further comprises providing the second image as an input to the machine learning model, wherein the iris segmentation data and pupil segmentation data most recently generated by the machine learning model are the third image 29. The method of example 28, comprising iris segmentation data and pupil segmentation data generated by a machine learning model for .

Example 32: The method of Example 28, wherein the head-mounted system is configured not to provide the second image as input to the machine learning model.

Example 33: A head-mounted system, a camera configured to capture an image of a user's eye, and in such a manner as to produce a flash of light captured by the camera in the image of the user's eye , a plurality of light sources configured to illuminate the eyes of a user, and one or more processors operably coupled to a camera, obtaining from the camera a first image of the eyes of the user; A neural network, which has been trained to generate iris segmentation data and pupil segmentation data given an image of the eye, is provided as input with a first image and a second image of the user's eye from the camera. A second image is captured by the camera immediately after the first image, each based on the iris segmentation data generated by the neural network with respect to the first image. detecting a set of one or more locations where one or more flashes of light are shown and, based on pupil segmentation data generated by the neural network for the first image, of a second image where the pupil of the user's eye is shown; identifying an area and determining an eye pose of the user based at least in part on the detected set of one or more flash locations in the second image and the identified area of the second image; A system comprising one or more processors configured to:

Example 34: The one or more processors further provide the second image as input to the neural network and obtain a third image of the user's eye from the camera, the third image being a representation of the second image. Detecting a set of one or more locations in a third image immediately captured by the camera, each showing one or more flashes based on iris segmentation data generated by the neural network for the second image. and identifying, based on pupil segmentation data generated by the neural network for the second image, a region of the third image in which the pupil of the user's eye is shown; 34. A system as in example 33, configured to determine a user's eye pose based on the detected set of one or more flash locations and the identified region of the third image.

Example 35: The one or more processors are further configured, based on iris segmentation data generated by the neural network for the first image, to render the second image in which the one or more flashes of light are shown. one or more processors, each configured to identify a specific region and based on iris segmentation data generated by the neural network for the first image, to detect a set of one or more locations; 34. A system as in example 33, wherein the system is configured to search for specific regions of the second image for a set of one or more locations where one or more flashes of light are shown.

Example 36: The one or more processors are further configured to avoid searching regions of the second image outside the specified region for the set of one or more locations where the one or more flashes of light are shown. , Example 35.

Example 37: The one or more processors are further configured to identify an iris contour of the user's eye in the first image based on iris segmentation data generated by the neural network for the first image. and based on the iris segmentation data generated by the neural network for the first image, the one or more processors determine the user's eye in the first image to identify a particular region of the second image. 36. The system of example 35, wherein the system is configured to identify the particular region of the second image based on the identified contour of the iris.

Example 38: The one or more processors are further configured to identify a centroid of the pupil of the user's eye based on pupil segmentation data generated by the neural network for the first image within the first image and based on the pupil segmentation data generated by the neural network for the first image, the one or more processors comprise a first 34. The system of example 33, wherein the system is configured to identify a region of the second image in which the pupil of the user's eye is shown based on the identified centroid of the pupil of the user's eye in the image of .

Example 39: Based on the identified centroid of the pupil of the user's eye in the first image, to identify a region of the second image where the pupil of the user's eye is shown, the one or more processors comprise , identifies a location in a second image based on the identified centroid of the pupil of the user's eye in the first image, and outwardly from the identified location in the second image, the pupil boundary. 39. The system of example 38, configured to search.

Example 40: The one or more processors in a starburst pupil detection routine to search outward from a location identified in the second image for the pupil boundary identified in the second image. 40. The system of example 39, configured to utilize location as a starting point.

Example 41: Determining a user's eye pose based, at least in part, on a detected set of one or more flash locations in a second image and an identified region of the second image To that end, the one or more processors determine the user's 34. The system of example 33, configured to obtain a position and orientation estimate of the optical axis of the eye.

Example 42: The one or more processors are further configured to estimate a three-dimensional location of the cornea of the user's eye in the second image based on the detected set of one or more flash locations in the second image. a value for a user's eye based, at least in part, on the detected set of one or more flash locations in the second image and the identified region of the second image; the one or more processors based at least in part on the estimated location of the cornea of the user's eye in the second image and the identified region of the second image to determine the pose of the 34. The system of example 33, wherein the system is configured to determine a user's eye pose.

Example 43: The one or more processors are further configured, based on the estimated location of the cornea of the user's eye in the second image, configured to obtain an estimate of a three-dimensional location of a pupil of a user's eye, and at least partially identifying an estimated location of a cornea of the user's eye in the second image and an identified location of the second image; the one or more processors, at least in part, the estimated location of the cornea of the user's eye in the second image; 43. The system of example 42, wherein the system is configured to determine the pose of the user's eye based on the estimated location of the pupil of the user's eye in the image of .

Example 44: The one or more processors further obtain from the camera a third image of the user's eye, the third image captured by the camera immediately after the second image and respectively processed by the neural network Detecting a set of one or more locations in the third image where one or more flashes are shown based on the most recently generated iris segmentation data and recently generated pupil segmentation data by the neural network. identifying a region of a third image in which the pupil of the user's eye is shown, based, at least in part, on the detected set of one or more flash locations in the third image; 34. The system of example 33, wherein the system is configured to determine the eye pose of the user based on the identified regions of the .

Example 45: The iris segmentation data and pupil segmentation data most recently generated by the neural network comprises the iris segmentation data and pupil segmentation data generated by the neural network for the second image to Example 44 System as described.

Example 46: The iris segmentation data and pupil segmentation data most recently generated by the neural network comprises the iris segmentation data and pupil segmentation data generated by the neural network for the first image to Example 44 System as described.

Example 47: The one or more processors are further configured to provide the second image as an input to the neural network, wherein the iris segmentation data and pupil segmentation data most recently generated by the neural network are used in a third 45. The system of example 44, comprising iris segmentation data and pupil segmentation data generated by the neural network for the image of .

[00410] Embodiment 48: The system of embodiment 44, wherein the one or more processors are further configured not to provide the second image as an input to the neural network.

As noted above, implementation of the illustrative examples provided above may include hardware, methods, or processes, and/or computer software on a computer-accessible medium.

Additional Considerations The processes, methods, and algorithms described herein and/or depicted in the accompanying figures are each configured to execute specific and specific computer instructions, one It may be embodied in code modules, and thereby fully or partially automated, executed by physical computing systems, hardware computer processors, application-specific circuits, and/or electronic hardware as described above. For example, a computing system can include a general purpose computer (eg, a server) or a special purpose computer, dedicated circuitry, etc. programmed with specific computer instructions. Code modules may be installed in a dynamically linked library, compiled and linked into an executable program, or written in an interpreted programming language. In some implementations, specific acts and methods may be performed by circuitry specific to a given function.

Moreover, any implementation of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex to require either special-purpose hardware (utilizing appropriate specialized executable instructions) or single-use hardware. More than one physical computing device may be required to perform the functionality, for example, due to the amount or complexity of computations involved, or to provide results substantially in real time. For example, a video may contain many frames, each frame having millions of pixels, and specifically programmed computer hardware can perform desired image processing tasks, There is a need to process video data to provide eye shape models, or biometric applications.

Code modules or any type of data may be stored in hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical discs, volatile or nonvolatile storage devices, combinations of the same, and/or It may be stored on any type of non-transitory computer-readable medium such as physical computer storage, including equivalents. The methods and modules (or data) may also be transmitted as data signals (e.g., carrier waves or other analog or digital propagated signals) generated on various computer-readable transmission media, including wireless-based and wire/cable-based media. part) and may take various forms (eg, as part of a single or multiplexed analog signal, or as a plurality of discrete digital packets or frames). The results of any disclosed process or process step can be persistently or otherwise stored in any type of non-transitory, tangible computer storage device or communicated over a computer-readable transmission medium.

Any process, block, state, step, or functionality in a flow diagram described herein and/or depicted in an accompanying figure represents a specific function (e.g., logic or arithmetic) or It should be understood as potentially representing a code module, segment, or portion of code containing one or more executable instructions for implementing a step. Various processes, blocks, states, steps, or functionality may be combined, rearranged, added to, deleted from, modified, or modified from the illustrative examples provided herein. It can be modified from there differently. In some embodiments, additional or different computing systems or code modules may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and blocks, steps, or states associated therewith may be performed in any other suitable sequence, e.g., serially, in parallel. or in some other manner. Tasks or events may be added to or removed from the disclosed exemplary embodiments. Moreover, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be appreciated that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged in multiple computer products. Many implementation variations are possible.

The present processes, methods and systems may be implemented in a network (or distributed) computing environment. Networking environments include enterprise-wide computer networks, intranets, local area networks (LAN), wide area networks (WAN), personal area networks (PAN), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. . The network can be a wired or wireless network or any other type of communication network.

The system and method of the present disclosure each have several innovative aspects, none of which are solely responsible or required for the desirable attributes disclosed herein. The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of this disclosure. Various modifications of the implementations described in this disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be adapted to other implementations without departing from the spirit or scope of this disclosure. can be applied. Accordingly, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the broadest scope consistent with the disclosure, principles and novel features disclosed herein. be.

Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Further, although features have been described above as acting in some combination and may also be originally claimed as such, one or more features from the claimed combination may, in some cases, be combined may be deleted from and claimed combinations may cover subcombinations or variations of subcombinations. No single feature or group of features is required or essential in every embodiment.

In particular, "can", "could", "might", "may", "eg (e.g.,)", and equivalents Conditional statements used herein, such as objects, generally refer to a feature, element, or feature of an embodiment, unless specifically stated otherwise or understood otherwise within the context in which they are used. , and/or steps, while other embodiments are intended to convey that they are not. Thus, such conditional statements generally state that features, elements, and/or steps are claimed in any way in one or more embodiments or that one or more embodiments necessarily include logic, with or without input or prompting, to determine whether these features, elements, and/or steps should be included or performed in any particular embodiment. not intended to agree. The terms “comprising,” “including,” “having,” and equivalents are synonymous and are used generically in a non-limiting manner, Do not exclude subjective elements, features, acts, actions, etc. Also, the term "or" is used in its inclusive sense (and not in its exclusive sense), thus, for example, when used to connect a list of elements, the term "or" Means one, some, or all of the elements in the list. Additionally, as used in this application and the appended claims, the articles "a," "an," and "the" refer to "one or more" or "at least one," unless specified otherwise. should be interpreted to mean

As used herein, a phrase referring to “at least one of” a list of items refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" encompasses A, B, C, A and B, A and C, B and C, and A, B, and C is intended. Conjunctive sentences, such as the phrase "at least one of X, Y, and Z," generally refer to at least one of X, Y, or Z, unless specifically stated otherwise. understood differently in contexts such as those used to convey that Thus, such conjunctions generally imply that certain embodiments require that at least one of X, at least one of Y, and at least one of Z each be present. is not intended to

Similarly, although operations may be depicted in the figures in a particular order, it is to be performed in the specific order in which such operations are shown or in a sequential order to achieve desirable results, or It should be appreciated that not all illustrated acts need be performed. Further, the drawings may graphically depict one or more exemplary processes in the form of flow charts. However, other acts not depicted may be incorporated within the illustrative methods and processes illustrated schematically. For example, one or more additional acts may be performed before, after, concurrently with, or between any of the illustrated acts. Additionally, the operations may be rearranged or reordered in other implementations. In some situations, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, as the described program components and systems are generally can be integrated together in multiple software products or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

Claims (32)

A head-mounted system comprising:
a camera configured to capture an image of a user's eye;
one or more light sources, wherein the one or more light sources are configured to illuminate the eye of the user such that a flash of light is represented in an image of the eye of the user; a light source;
one or more processors, the one or more processors comprising:
obtaining a first image of the user's eye from the camera;
providing the first image as input to a machine learning model that has been trained to generate iris segmentation data and pupil segmentation data given an image of the eye;
obtaining from the camera a second image of the user's eye following the first image;
detecting a set of one or more locations in the second image where one or more flashes are represented based on the iris segmentation data;
identifying a region of the second image in which a pupil of the user's eye is represented based on the pupil segmentation data;
Determining the user's eye pose based, at least in part, on the detected set of one or more flash locations in the second image and the identified region of the second image. A system comprising one or more processors configured to perform The one or more processors further:
obtaining iris segmentation data and pupil segmentation data for the second image via the machine learning model;
obtaining a third image of the user's eye from the camera;
detecting a set of one or more locations in the third image where one or more flashes are indicated, each based on the iris segmentation data generated by the machine learning model for the second image; and
identifying a region of the third image in which a pupil of the user's eye is represented based on the pupil segmentation data generated by the machine learning model for the second image;
Determining the user's eye pose based, at least in part, on the detected set of one or more flash locations in the third image and the identified region of the third image. 2. The system of claim 1, configured to perform and. The one or more processors further:
configured to identify a second region of the second image based on the iris segmentation data;
To detect a set of one or more locations in the second image where one or more flashes of light are shown, the one or more processors:
2. The system of claim 1, configured to search the second region of the second image for distinct locations exhibiting maximal intensity values within the second region. The one or more processors further:
4. The system of claim 3, configured to avoid searching regions of the second image outside the specified region for a set of one or more locations where one or more flashes of light are shown. The one or more processors further:
configured to identify an iris contour of the user's eye in the first image based on the iris segmentation data;
To identify different regions of the second image based on the iris segmentation data, the one or more processors:
4. The system of claim 3, configured to identify a third region of the second image based on identified contours of the iris of the user's eye in the first image. The one or more processors further:
based on the pupil segmentation data, identify the centroid of the pupil of the user's eye in the first image;
To identify regions of the second image, the one or more processors:
based on the identified centroid of the pupil of the user's eye in the first image, identifying a second region of the second image in which the pupil of the user's eye is represented. , the system of claim 1. To identify a second region of the second image, the one or more processors:
identifying a location in the second image based on the identified centroid of the pupil of the user's eye in the first image;
7. The system of claim 6, configured to: retrieve pupillary boundaries from locations identified in the second image. To retrieve pupillary boundaries from locations identified in the second image, the one or more processors:
8. The system of claim 7, configured to perform a starburst pupil detection process based on locations identified in the second image that are assigned as starting points. for determining an eye pose of the user based, at least in part, on a detected set of one or more flash locations in the second image and identified regions of the second image; and the one or more processors are:
Based, at least in part, on the detected set of one or more flash locations in the second image and the identified region of the second image, determining the position of the optical axis of the user's eye and 2. The system of claim 1, configured to obtain an orientation estimate. The one or more processors further:
obtain a three-dimensional location estimate of the cornea of the user's eye in the second image based on a detected set of one or more flash locations in the second image;
To determine the user's eye pose, the one or more processors:
determining the pose based, at least in part, on an estimated location of the cornea of the user's eye in the second image and an identified region of the second image. , the system of claim 1. The one or more processors further:
3 of the pupil of the user's eye in the second image based on the estimated location of the cornea of the user's eye in the second image and the identified region of the second image; configured to obtain a dimensional location estimate,
To determine the user's eye pose, the one or more processors:
The pose is based, at least in part, on an estimated location of a cornea of the user's eye within the second image and an estimated location of a pupil of the user's eye within the second image. 11. The system of claim 10, configured to determine: The one or more processors further:
obtaining a third image of the user's eye;
detecting a set of one or more locations in the third image where one or more flashes of light are represented, each based on iris segmentation data most recently generated by the machine learning model;
identifying a region of the third image where the pupil of the user's eye is shown based on pupil segmentation data most recently generated by the machine learning model;
determining a second pose of the user's eye based, at least in part, on the detected set of one or more flash locations in the third image and the identified region of the third image; 2. The system of claim 1, configured to: determine; 3. The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprise iris segmentation data and pupil segmentation data generated by the machine learning model for the second image. 13. The system according to 12. 3. The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprise iris segmentation data and pupil segmentation data generated by the machine learning model for the first image. 13. The system according to 12. The one or more processors further:
configured to provide the second image as an input to the machine learning model;
3. The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprise iris segmentation data and pupil segmentation data generated by the machine learning model for the third image. 13. The system according to 12. 13. The system of claim 12, wherein the one or more processors are further configured to avoid providing the second image as input to the machine learning model. A method implemented by a head-mounted system of one or more processors, wherein the head-mounted system is configured to illuminate an eye of a user such that a flash of light is represented in an image of the eye. and the method includes:
obtaining a first image of the user's eye;
providing the first image as input to a machine learning model that has been trained to generate iris segmentation data and pupil segmentation data given an image of the eye;
obtaining a second image of the user's eye;
detecting a set of one or more locations in the second image where one or more flashes are represented based on the iris segmentation data;
identifying a region of the second image in which a pupil of the user's eye is represented based on the pupil segmentation data;
Determining the user's eye pose based, at least in part, on the detected set of one or more flash locations in the second image and the identified region of the second image. A method comprising and . obtaining iris segmentation data and pupil segmentation data for the second image via the machine learning model;
obtaining a third image of the user's eye;
detecting a set of one or more locations in the third image where one or more flashes are indicated, each based on the iris segmentation data generated by the machine learning model for the second image; and
identifying a region of the third image in which a pupil of the user's eye is represented based on the pupil segmentation data generated by the machine learning model for the second image;
Determining the user's eye pose based, at least in part, on the detected set of one or more flash locations in the third image and the identified region of the third image. 18. The method of claim 17, further comprising: and. identifying a second region of the second image based on the iris segmentation data;
For detecting a set of one or more locations in the second image where one or more flashes are shown, the method further comprises
18. The method of claim 17, comprising searching a second region of the second image for distinct locations exhibiting maximum intensity values within the second region. The head-mounted system is configured to avoid searching regions of the second image outside the specified region for a set of one or more locations where one or more flashes of light are shown. Item 20. The method according to Item 19. identifying an iris outline of the user's eye in the first image based on the iris segmentation data;
To identify different regions of the second image based on the iris segmentation data, the method further comprises:
20. The method of claim 19, comprising identifying a third region of the second image based on identified contours of the iris of the user's eye in the first image. identifying the centroid of the pupil of the user's eye in the first image based on the pupil segmentation data;
To identify regions of the second image, the method further comprises:
identifying a second region of the second image in which the pupil of the user's eye is represented based on the identified centroid of the pupil of the user's eye in the first image. Item 18. The method according to Item 17. To identify a second region of the second image, the method comprises:
identifying a location in the second image based on the identified centroid of the pupil of the user's eye in the first image;
23. The method of claim 22, comprising retrieving pupillary boundaries from locations identified in the second image. For retrieving a pupillary boundary from a location identified in the second image, the method comprises:
24. The method of claim 23, comprising performing a starburst pupil detection process based on locations identified in the second image that are assigned as starting points. for determining an eye pose of the user based, at least in part, on a detected set of one or more flash locations in the second image and identified regions of the second image; In addition, the method includes:
Based, at least in part, on the detected set of one or more flash locations in the second image and the identified region of the second image, determining the position of the optical axis of the user's eye and 18. The method of claim 17, comprising obtaining an orientation estimate. The method includes
obtaining a three-dimensional location estimate of the cornea of the user's eye in the second image based on the detected set of one or more flash locations in the second image;
To determine the user's eye pose, the method comprises:
determining the pose based, at least in part, on an estimated location of the cornea of the user's eye in the second image and an identified region of the second image. Item 18. The method according to Item 17. The method further comprises:
3 of the pupil of the user's eye in the second image based on the estimated location of the cornea of the user's eye in the second image and the identified region of the second image; obtaining an estimate of the dimensional location,
To determine the user's eye pose, the method comprises:
The pose is based, at least in part, on an estimated location of a cornea of the user's eye within the second image and an estimated location of a pupil of the user's eye within the second image. 27. The method of claim 26, comprising determining: The method further comprises:
obtaining a third image of the user's eye;
detecting a set of one or more locations in the third image where one or more flashes of light are represented, each based on iris segmentation data most recently generated by the machine learning model;
identifying a region of the third image where the pupil of the user's eye is shown based on pupil segmentation data most recently generated by the machine learning model;
determining a second pose of the user's eye based, at least in part, on the detected set of one or more flash locations in the third image and the identified region of the third image; 18. The method of claim 17, comprising determining; 3. The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprise iris segmentation data and pupil segmentation data generated by the machine learning model for the second image. 28. The method according to 28. 3. The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprise iris segmentation data and pupil segmentation data generated by the machine learning model for the first image. 28. The method according to 28. The method further comprises:
providing the second image as an input to the machine learning model;
3. The iris segmentation data and pupil segmentation data most recently generated by the machine learning model comprise iris segmentation data and pupil segmentation data generated by the machine learning model for the third image. 28. The method according to 28. 29. The method of claim 28, wherein the head-mounted system is configured not to provide the second image as input to the machine learning model.

Images