KR20230092834A - Eye tracking using self-mixing interferometry - Google Patents

Abstract

An eye tracking device includes a head-mountable frame, an optical sensor subsystem mounted to the head-mountable frame, and a processor. The optical sensor subsystem includes a set of one or more SMI sensors. The processor operates an optical sensor subsystem to cause a set of one or more SMI sensors to emit a set of one or more beams of light toward an eye of a user; receive a set of one or more SMI signals from a set of one or more SMI sensors; and track eye movement using a set of one or more SMI signals.

Description

Eye tracking using self-mixing interferometry {EYE TRACKING USING SELF-MIXING INTERFEROMETRY}

CROSS REFERENCES TO RELATED APPLICATIONS

[0001] This application is a non-provisional application and claims 35 U.S.C. The benefit under §119(e) is claimed, the contents of which are incorporated herein by reference.

technology field

The described embodiments relate generally to optical sensing, and more specifically to eye movement tracking using optical sensors.

Eye monitoring technologies may be used to improve near eye displays ( eg , head-mounted displays (HMDs)), augmented reality (AR) systems, virtual reality (VR) systems, and the like. For example, gaze vector tracking, also known as gaze position tracking, can be used as an input for display fovea rendering or human-computer interaction. Conventional eye monitoring techniques are camera-based or video-based, and rely on active illumination of the eye, eye image acquisition, and extraction of eye features such as pupil center and corneal reflection. The power consumption, form factor, computational cost, and latency of these eye monitoring technologies make them more user-friendly next-generation HMD, AR, and VR systems ( e.g. , lighter, battery-operated, more full-featured systems). can be a significant burden on

Embodiments of systems, devices, methods, and apparatus described in this disclosure utilize one or more self-mixing interferometry (SMI) sensors to track eye movement. In some embodiments, SMI sensors are used alone or in combination with a camera to also determine a gaze vector or eye position.

In a first aspect, the present disclosure describes an eye tracking device. An eye tracking device may include a head-mountable frame, an optical sensor subsystem mounted to the head-mountable frame, and a processor. The optical sensor subsystem may include a set of one or more SMI sensors. The processor operates an optical sensor subsystem to cause a set of one or more SMI sensors to emit a set of one or more beams of light toward an eye of a user; receive a set of one or more SMI signals from a set of one or more SMI sensors; It may be configured to track eye movement using a set of one or more SMI signals.

In a second aspect, the present disclosure describes another eye tracking device. An eye tracking device may include a set of one or more SMI sensors, a camera, and a processor. The processor causes the camera to obtain, at a first frequency, a set of images of the user's eye; cause a set of one or more SMI sensors to emit a set of one or more beams of light toward an eye of a user; sampling a set of one or more SMI signals generated by a set of one or more SMI sensors at a second frequency greater than the first frequency; determining a gaze vector of an eye using at least one image in the set of images; It may be configured to track eye movement using a set of one or more SMI signals.

In a third aspect, the present disclosure describes a method of tracking eye movement. The method includes operating an optical sensor subsystem to cause a set of one or more SMI sensors in the optical sensor subsystem to emit one or more sets of beams of light toward an eye of a user; receiving a set of one or more SMI signals from a set of one or more SMI sensors; and tracking eye movement using the set of one or more SMI signals.

In addition to the foregoing exemplary aspects and embodiments, further aspects and embodiments will become apparent by reference to the drawings and by a study of the description below.

BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure will be readily understood by the following detailed description taken in conjunction with the accompanying drawings, in which like reference numbers indicate like structural elements.
1 shows an exemplary block diagram of an eye tracking device.
2A shows an example graph of tracked angular velocity of the eye.
2B shows an example graph of an eye's tracked gaze.
3A shows a first example eye-tracking device with an optical sensing subsystem and processor mounted on eyeglasses.
3B shows a second exemplary eye-tracking device with an optical sensing subsystem and processor mounted in a VR headset.
4A shows a side view of an example set of SMI sensors that can be mounted to a head-mountable frame and configured to emit light toward an eye.
FIG. 4b shows a front view of the set of eye and SMI sensors shown in FIG. 4a.
5 shows an exemplary front view of a first alternative set of SMI sensors that can be mounted to a head-mountable frame and configured to emit light toward the eye.
FIG. 6 shows an exemplary front view of a second alternative set of SMI sensors that can be mounted to a head-mountable frame and configured to emit light toward an eye.
FIG. 7 shows an exemplary side view of a third alternative set of SMI sensors that can be mounted to a head-mountable frame and configured to emit light toward an eye.
8A shows an exemplary use of an SMI sensor in combination with a beam splitter.
8B shows an example use of an SMI sensor in combination with a beam steering component.
9A shows a first exemplary integration of an SMI sensor with a display subsystem.
9B shows a second exemplary integration of the SMI sensor with the display subsystem.
10 shows an example set of components that may be included in an optical sensor subsystem of an eye tracking device.
11A shows a first example method for tracking eye movement using a set of one or more SMI sensors.
11B illustrates a second example method for tracking eye movement using a set of one or more SMI sensors in combination with a camera or other sensors.
12A and 12B show how one or more surfaces or structures of the eye may be mapped using one or more sets of SMI sensors.
The use of cross-hatching or shading in the accompanying drawings is generally provided to clarify boundaries between adjacent elements and to facilitate readability of the drawings. Accordingly, the presence or absence of cross-hatching or shading may not be relevant to a particular material, material properties, element proportions, element dimensions, commonalities of like-illustrated elements, or any other characteristic relative to any element shown in the accompanying drawings; No preference or requirement for properties or attributes is implied or indicated.
Further, the proportions and dimensions (relative or absolute) of the various features and elements (and assemblies and groups thereof) and the boundaries, separations, and positional relationships provided therebetween are only shown in the accompanying drawings. It is provided to facilitate an understanding of the various embodiments described herein, and therefore may not necessarily be drawn or drawn to scale, and make no claims about the illustrated embodiments, excluding the embodiments described with reference thereto. It should be understood that neither preference nor requirement is intended to be indicated.

Reference will now be made in detail to representative embodiments illustrated in the accompanying drawings. It should be understood that the following description is not intended to limit the embodiments to one preferred embodiment. On the contrary, it is intended to cover alternatives, modifications and equivalents as may be included within the spirit and scope of the described embodiments as defined by the appended claims.

Most eye tracking systems are camera or image based and cannot track eye movement fast enough or with high enough accuracy, using an adequate amount of power. Less power and less costly systems are needed, with higher accuracy and sensitivity.

Although rapid and accurate detection and classification of eye movements, such as distinguishing between smooth pursuit, saccade, fixation, nystagmus, and blinking, can be an important task, It can also be difficult for camera-based, video-based, or photodetector-based tracking systems - especially under tight power budgets. For example, a gaze can be as short as tens of microseconds and imperceptible as <0.25 degrees per second (deg/s) or <0.5 deg/s. Detection of these eye movements suggests a need for high resolution and high frame rate image acquisition and processing systems.

Previously, reduced resolution and higher frame rate image captures were performed using higher resolution and lower frame rate image captures for overall power savings ( e.g. , compared to systems that only acquire higher resolution images at higher frame rates). A low-power imaging-based eye odometry fused with captures has been proposed. Alternatively, photodiode array-based eye trackers with lower latency and lower power consumption have been proposed, but have not yet proven useful in higher resolution and higher accuracy applications. Photodiode array-based eye trackers are therefore more suitable for binary sensing applications, such as mission critical video-based eye trackers, or wake up near eye display (or HMD) systems that rely on sparse gaze zone detection or gaze movement thresholding. Suitable.

The description below relates to eye movement tracking using SMI sensors - alone or in combination with other sensors, such as a camera or other image-based sensor. For purposes of this description, an SMI sensor includes a light emitter ( e.g. , a laser light source) and an SMI signal detector ( e.g. , a photodetector such as a photodiode, or an electrical detector such as a circuit that measures the junction voltage or drive current of the light emitter). ) is considered to include Each of the one or more SMI sensors emits one or more fixed or scanned beams of light towards one or more structures of the user's eye ( eg, towards one or more of the eye's iris, sclera, pupil, lens, limbus, eyelid, etc.) can do. The SMI sensor(s) may be operated in compliance with operational safety specifications so as not to harm the user's eyes.

After emitting a beam of light, the SMI sensor can receive the retroreflected portion of the emitted light back into its resonant cavity. For good quality retroreflection, focusing the beam of light emitted by the SMI sensor on the iris or sclera (or other diffusing structure) of the eye may be useful compared to focusing the beam of light on the cornea or pupil of the eye. . The phase change of the retroreflected portion of the emitted light can be mixed with the phase of the light produced within the resonant cavity and can produce an SMI signal that can be amplified and detected. An amplified SMI signal, and in some cases multiple SMI signals, may be analyzed. The rotational movement of the eye can be retrieved and reconstructed by phase tracking the Doppler frequencies from multiple orientations and positions of the user's eye.

Classification and quantification of user gaze behaviors such as euphemism, saccade, gaze, nystagmus, and blink are identified at high sampling rates using SMI sensors, resulting in high levels of numbers, text, or images on a near eye display (or HMD) system. It can enable efficient, high-fidelity, digital content rendering.

In some embodiments, the SMI sensor data or determinations are fused with absolute gaze direction sensing information ( eg, gaze vector detection or gaze position detection) obtained from a lower sampling rate gaze imaging system ( eg , camera) to obtain an even higher It can enable absolute eye tracking with speed and excellent accuracy. In contrast to an imaging system that may include a million or more pixels, SMI sensor data may be obtained from one or a small number ( eg , two or three) SMI sensors. This may allow SMI sensors to generate SMI signals (or SMI sensor data) with much lower power consumption compared to an imaging system.

When the wavelength of the beam of light emitted by the SMI sensor is modulated, the SMI signal obtained from the SMI sensor, at a level of ~100 μm level resolution, can be used for absolute ranging of surfaces, interfaces, and volume structures of the user's eye. can This absolute distance measurement can provide a fixation point for tracking the displacement of the eye profile during eye rotation.

When the wavelength(s) of the beam(s) of light emitted by one or more SMI sensors is modulated, and when the beam(s) of light emitted by at least one SMI sensor is scanned and/or multiple beams are emitted, Doppler Displacement and/or velocity maps, also known as clouds, and/or distance maps, also known as depth clouds, may be obtained or constructed. Additionally or alternatively, a Doppler cloud may be obtained or constructed when the wavelength(s) of the beam(s) of light emitted by one or more SMI sensors are not modulated. The Doppler cloud may inherently have a high resolution, which may be, for example, on the μm or sub-μm level for a single frame displacement, or at least on the mm/s or deg/s level for velocity. A single or multiple frames of a Doppler cloud can be considered a differential depth cloud. Additionally or alternatively, measurements of single or multiple frames of the Doppler cloud match a predefined and/or locally calibrated difference map or library and eye tracking information and/or positional information (also known as posing information) It can be processed in real time to extract . A locally calibrated difference map or library may be obtained by means including, but not limited to, cameras, depth clouds, and the like. Using a Doppler cloud, alone or fused with a depth cloud or other sensing modalities ( eg , eye camera images, motion sensors, etc.), can provide an accurate and efficient way to track eye movement or positional information.

These and other systems, devices, methods and apparatus are described with reference to FIGS. 1-12B. However, those skilled in the art will readily appreciate that the detailed description provided herein with respect to these drawings is for explanatory purposes only and should not be construed as limiting.

Directions such as "up", "down", "top", "bottom", "front", "back", "up", "down", "up", "down", "left", "right", etc. The term is used in relation to the orientation of some of the components in some of the figures described below. Since components of various embodiments may be positioned in a number of different orientations, directional terms are used for descriptive purposes only and in no way limiting. Orientation terms are intended to be interpreted broadly and, therefore, should not be construed to exclude that components may be oriented in different ways. As used herein, the phrase "at least one of" preceding a series of items, with the terms "and" or "or" separating any of the items, rather than each member of the list, Fully modifies the list. The phrase "at least one of" does not require selection of at least one of each item listed; Rather, the phrase permits a meaning that includes at least one of any of the items, and/or at least one of any combination of items, and/or at least one of each of the items. By way of example, the phrase “at least one of A, B, and C” or the phrase “at least one of A, B, or C” may each include only A, only B, or only C; any combination of A, B, and C; and/or one or more of each of A, B, and C. Similarly, it can be understood that the order of elements presented for associativity or separability lists provided herein should not be construed as limiting the present disclosure only to the order presented.

1 shows an exemplary block diagram of an eye tracking device 100 . The eye tracking device 100 may include a head-mountable frame 102 , an optical sensor subsystem 104 mounted to the head-mountable frame 102 , and a processor 106 . In some embodiments, eye-tracking device 100 also includes one of display subsystem 108, communication subsystem 110 ( eg , wireless and/or wired communication subsystem), and power distribution subsystem 112. may contain more than Processor 106, display subsystem 108, communication subsystem 110, and power distribution subsystem 112 are either partially or wholly mounted on head-mountable frame 102, or head-mountable frame Radio frequency or electrical communication with one or more components mounted on 102 ( e.g. , radio frequency ( i.e. , wireless) or electrical ( e.g. , cord connection) with one or more components mounted on head-mountable frame 102. ) housed within a box or electronic device ( e.g. , a phone, or a wearable device such as a watch) that communicates with the head-mountable frame 102 and one or more components mounted to the head-mountable frame 102. It can be distributed between boxes or electronic devices with radio frequency or electrical communication. Optical sensor subsystem 104, processor 106, display subsystem 108, communication subsystem 110, and/or power distribution subsystem 112 may use one or more communication protocols to communicate with one or more buses. 116, or through the air ( eg , wirelessly).

The head-mountable frame 102 may take the form of glasses, a set of goggles, an augmented reality (AR) headset, a virtual reality (VR) headset, or other forms of the head-mountable frame 102 .

Optical sensor subsystem 104 may include a set of one or more SMI sensors 114 . Each SMI sensor in the set of one or more SMI sensors can include a light emitter and a light detector. The light emitter may be a vertical-cavity surface-emitting laser (VCSEL), an edge-emitting laser (EEL), a vertical external-cavity surface-emitting laser (VECSEL), a quantum-dot laser (QDL), a quantum cascade laser (QCL), or light emitting diodes (LEDs) ( e.g. , organic LEDs (OLEDs), resonant-cavity LEDs (RC-LEDs), micro LEDs (mLEDs), high brightness LEDs (SLEDs), or edge-emitting LEDs), and the like). there is. A photodetector (or photodetector) may in some cases be positioned laterally adjacent to the light emitter ( eg , mounted or formed on a substrate on which the photodetector is mounted or formed). In other cases, the light detector may be stacked above or below the light emitter. For example, the photodetector can be a VCSEL, HCSEL, or EEL with a primary emission and a secondary emission, the photodetector is epitaxially formed of the same epitaxial stack as the photoemitter, so that the photodetector can generate some or all of the secondary emission. can be made to receive light. In these latter embodiments, the light emitter and light detector may be formed similarly ( e.g. , both the light emitter and light detector may include multiple quantum well (MQW) structures, but the light emitter may be forward biased). and the photodetector ( eg , a resonant cavity photodetector (RCPD)) can be reverse biased. Alternatively, the light detector can be formed on the substrate and the light emitter can be separately formed and mounted on the substrate, or formed and mounted in conjunction with the light emitter so that the secondary light emission of the light emitter impinges the light detector. can Alternatively, the light emitter can be formed on the substrate and the light detector can be separately formed and mounted on the substrate, or formed and mounted in conjunction with the light emitter so that the secondary light emission of the light emitter impinges the light detector. can

In some embodiments, optical sensor subsystem 104 includes a set of fixed or movable optical components ( eg , one or more lenses, optical gratings, optical filters, beam splitters, beam steering components, etc.) can do. Optical sensor subsystem 104 may also include an image sensor ( eg , a camera including an image sensor).

Processor 106 may include any electronic device capable of processing, receiving, or transmitting data or instructions, whether such data or instructions are in the form of software or firmware, or otherwise encoded. For example, processor 106 may include a microprocessor, central processing unit (CPU), application specific integrated circuit (ASIC), digital signal processor (DSP), controller, or a combination of these devices. As used herein, the term “processor” is intended to encompass a single processor or processing unit, multiple processors, multiple processing units, or other suitably configured computing elements.

In some embodiments the components of eye tracking device 100 may be controlled by multiple processors. For example, select components of the eye-tracking device 100 ( eg , the optical sensor subsystem 104) may be controlled by the first processor, and other components of the eye-tracking device ( eg , the display subsystem ( 108) and/or communication subsystem 110) may be controlled by a second processor, and the first and second processors may or may not be in communication with each other.

In some embodiments, display subsystem 108 has one or more light emitting elements including, for example, an LED, OLED, liquid crystal display (LCD), electroluminescent (EL) display, or other types of display elements. may include a display.

Communications subsystem 110 may enable eye tracking device 100 to transmit or receive data from a user or other electronic device. Communications subsystem 110 may include a touch-sensitive input surface, a crown, one or more microphones or speakers, or a wired or wireless ( eg , radio frequency (RF) or optical) communication interface configured to transmit electronic, RF, or optical signals. can include Examples of wireless and wired communication interfaces include, but are not limited to, cellular, Wi-Fi, and Bluetooth® communication interfaces.

Power distribution subsystem 112 may be implemented with any collection of power sources and/or conductors capable of delivering energy to eye tracking device 100 or components thereof. In some cases, power distribution subsystem 112 may include one or more batteries or rechargeable batteries. Additionally or alternatively, power distribution subsystem 112 may connect eye tracking device 100 or components thereof to a remote power source, such as a wall outlet, a remote battery pack, or an electronic device to which eye tracking device 100 is tethered. It may include a power connector or power cord that can be used to connect to.

Processor 106 may be configured to operate optical sensor subsystem 104 . Operating the optical sensor subsystem 104 causes the power distribution subsystem 112 to power the optical sensor subsystem 104 and provide control signals to the set of one or more SMI sensors 114 and/or or providing control signals that electrically, electromechanically, or otherwise focus or tune the optical components of the optical sensor subsystem 104 . Operating the optical sensor subsystem 104 may cause the set of one or more SMI sensors 114 to emit sets of one or more beams of light 118 towards the eye 120 of the user. Processor 106 may also be configured to receive a set of one or more SMI signals from a set of one or more SMI sensors 114 and use the set of one or more SMI signals to track rotational movement of eye 120 .

In some cases, optical sensor subsystem 104, processor 106, display subsystem 108, communication subsystem 110, and/or power distribution subsystem 112 may communicate over one or more buses. may be, which is generally described as bus 116.

Tracking the rotational movement of the eye 120 may in some cases include estimating the angular velocity (or gaze movement) of the eye 120 . In some cases, angular velocity can be tracked in each of three orthogonal directions ( eg , in the x, y, and z directions). Tracking the angular velocities in different directions scans one or more beams of light emitted by a set of one or more SMI sensors 114 and emitted by one or more SMI sensors in the set of one or more SMI sensors 114. splitting one or more beams of light, or configuring a set of one or more SMI sensors 114 to emit two or more beams of light in different directions (and preferably in different orthogonal directions). there is.

Tracking the rotational movement of the eye 120 may also include tracking the gaze (or gaze position) of the eye 120 .

In some embodiments, the processor 106 may be configured to classify the user's rotational eye movement. For example, rotational eye movements may be classified as at least one of euphemism, saccade, gaze, nystagmus, or blink. Processor 106 may then cause display subsystem 108 to adjust the rendering of one or more images on the display in response to the classification of the rotational eye movement. In some embodiments, the processor 106 is configured to quantify the classified rotational eye movement ( e.g. , how fast or how much the user blinks, what is the angular velocity of the user's eye movement, or the angular velocity of the user's eye 120). how frequent or how fast the change is), and the rendering of one or more images can be further adjusted in response to the quantified rotational eye movement.

In some embodiments, processor 106 may be configured to cause display subsystem 108 to change the state of the display in response to eye 120 movement. Changing the state of the display may include changing what is displayed, but also or alternatively, transitioning the display from a low power or OFF state to a higher power or ON state, or alternatively, switching the display to a higher power state. Or it may include switching from an ON state to a low power or OFF state.

In some embodiments, the display may include an array of display pixels mounted on a substrate, and a set of one or more SMI sensors 114 may be mounted on the substrate adjacent to or within the array of display pixels at least It can include one SMI sensor. In other embodiments, the SMI sensor(s) 114 may be provided remotely from the display.

2A shows an example graph 200 of tracked angular velocity of the eye (or gaze movement). As an example, graph 200 shows traces of angular velocity in only two orthogonal directions. When the outer surface of the eye or other structure is modeled as a two-dimensional object, the angular velocity can only be tracked in two dimensions. Alternatively, the angular velocity can be tracked more precisely in three dimensions.

2B shows an example graph 210 of an eye's tracked gaze (or gaze vector, or gaze position). As an example, graph 210 shows gaze tracking in only two orthogonal directions. When the outer surface or other structure of the eye is modeled as a two-dimensional object, the gaze can only be tracked in two dimensions. Alternatively, the line of sight can be tracked more precisely in three dimensions.

SMI-based sensing can be particularly useful when a device or application needs to detect and/or classify subtle eye movements such as euphemism, saccade, gaze, nystagmus, and blink, since all such eye movements They can be detected and classified by tracking the eye's angular velocity, and SMI-based sensing is typically faster, more accurate, and more power-efficient than video-based or photodetector-based sensing when it comes to detecting changes in the eye's angular velocity. Because it is efficient ( eg , consumes less power). Also, with initial and periodic (low frequency) correction of gaze position, high frequency gaze position vector tracking can be obtained from integration of gaze velocity vector tracking, as shown in FIG. 2B , as shown in FIG. 2A .

FIG. 3A shows a first example eye-tracking device in which an optical sensing subsystem 302 and a processor 304 are mounted to eyeglasses 300 . As an example, eyeglasses 300 ( i.e. , a type of head-mountable frame) may include a first lens rim 308 and a second lens rim 310, the first lens rim 308 coupled to the second lens rim 310. a bridge 312 connected to the first lens rim 308, a first temple 314 connected to the lens rim 308, and a second temple 316 connected to the second lens rim 310. In some embodiments, glasses 300 may include a heads-up display or function as AR glasses.

Each of the first lens rim 308 and the second lens rim 310 may hold a respective lens, such as a first lens 318 or a second lens 320 . Lenses 318 and 320 may or may not magnify, focus, or otherwise alter light passing through lenses 318 and 320 . For example, lenses 318 and 320 may correct a user's vision, block bright or harmful light, or simply provide a physical barrier through which light may pass with no or minimal adjustments. there is. In some embodiments, first and second lenses 318 and 320 may be formed of glass or plastic. In some embodiments, first and/or second lenses 318, 320 function as a display ( eg , a passive display screen) onto which text, numbers, and/or images are projected by display subsystem 322. and the display subsystem 322 can also be mounted on eyeglasses. Alternatively, the first and/or second lens rims 308, 310 may display text, digits, or text by a transparent or translucent display ( eg , light emitting diode (LED), organic LED (OLED), or display subsystem 322). , and/or other light emitting elements operable to display images.

As a further example, optical sensing subsystem 302 may be configured as described with reference to one or more of FIGS. 1 and 5-9B and/or processor 304 may be configured with respect to FIGS. It may be configured to operate as described with reference. One or more components ( e.g. , SMI sensors, optical components, camera, etc.) of optical sensing subsystem 302 include a first lens rim 308, a second lens rim 310, a bridge 312, a first Can be mounted on one or more substrates attached to temple 314, second temple 316, first lens 318, or second lens 320, or mounted directly to one or more of these components It can be. Similarly, processor 304, display subsystem 322, communication subsystem 324, and/or power distribution subsystem 326 may be incorporated into one or more of these components. In some embodiments, some or all of the optical sensing subsystem 302, processor 304, display subsystem 322, communication subsystem 324, and/or power distribution subsystem 326 may include glasses ( 300), within a device wirelessly or electrically connected to one or more components mounted on the glasses 300 ( e.g. , a user's phone or wearable device), or between the glasses 300 and the glasses ( 300) may be distributed among devices wirelessly or electrically connected to one or more components.

Processor 304 , display subsystem 322 , communication subsystem 324 , and power distribution subsystem 326 may be further configured or operated as described with reference to FIG. 1 .

3B shows a second exemplary eye-tracking device in which an optical sensing subsystem 352 and a processor 354 are mounted to a virtual reality (VR) headset 350 . As an example, a VR headset 350 ( ie , a type of head-mountable frame) is shown to include a VR module 358 that can be attached to the user's head on a strap 356.

VR module 358 may include display subsystem 360 . Display subsystem 360 may include a display for displaying text, numbers, and/or images.

As an example, optical sensing subsystem 352 may be configured as described with reference to one or more of FIGS. 1 and 5-9B and/or processor 354 may be configured with reference to FIGS. 1 and 10-12B. may be configured to operate as described above. One or more components of the optical sensing subsystem 352 ( eg , SMI sensors, optical components, camera, etc.) can be mounted on one or more substrates attached to the VR module 358, or the VR module ( 358) can be mounted directly on the housing. Similarly, processor 354 , display subsystem 360 , communication subsystem 362 , and/or power distribution subsystem 364 may be incorporated into VR module 358 . In some embodiments, some or all of optical sensing subsystem 352, processor 354, display subsystem 360, communication subsystem 362, and/or power distribution subsystem 364 may be part of a VR module. mounted within a device wirelessly or electrically connected to 358 ( e.g. , within a user's phone or wearable device), or distributed between VR module 358 and a device wirelessly or electrically connected to VR module 358. can

Processor 354 , display subsystem 360 , communication subsystem 362 , and power distribution subsystem 364 may be further configured or operated as described with reference to FIG. 1 .

4A and 4B show an exemplary set of SMI sensors 400, 402 that can be mounted to a head-mountable frame, such as one of the head-mountable frames described with reference to FIGS. 1-3B. do. The set of SMI sensors 400, 402 may form part of an optical sensor subsystem, such as one of the optical subsystems described with reference to FIGS. 1-3B or elsewhere herein. A set of SMI sensors 400 and 402 may be configured to emit light toward eye 404 . 4A shows a side view of an eye 404 and a set of SMI sensors 400, 402, and FIG. 4B shows a front view of an eye 404 and a set of SMI sensors 400, 402.

As an example, a set of SMI sensors 400, 402 is shown in FIGS. 4A and 4B to include two SMI sensors ( eg , first SMI sensor 400 and second SMI sensor 402). In other embodiments, the set of SMI sensors 400, 402 may include more or fewer SMI sensors. The first SMI sensor 400 and the second SMI sensor 402 may emit respective beams of light toward the eye 404 . In some embodiments, the beams of light may be directed in different directions, which may or may not be orthogonal directions. If the beams of light are directed in different directions, a processor receiving the SMI signals generated by the SMI sensors 400 and 402 can track the movement of the eye 404 in two dimensions ( e.g. , the processor can can track the angular velocities of the eye 404 at . The beams of light may impinge on the eye 404 at the same or different locations. As an example, beams of light are shown impinging on the eye 404 at the same location.

In some embodiments, light emitted by SMI sensors 400, 402 is directed by optics 406 or 408 ( eg , one or more lenses or beam steering components or other optics components). or filtered.

5 is a front view of an alternative set of SMI sensors 500, 502, 504 that can be mounted to a head-mountable frame, such as one of the head-mountable frames described with reference to FIGS. 1-3B. show A set of SMI sensors 500, 502, 504 may form part of an optical sensor subsystem, such as one of the optical subsystems described with reference to FIGS. 1-3B or elsewhere herein. A set of SMI sensors 500 , 502 , 504 may be configured to emit light towards eye 506 .

In contrast to the set of SMI sensors described with reference to FIGS. 4A and 4B , the set of SMI sensors 500, 502, and 504 shown in FIG. 5 includes three SMI sensors ( e.g. , the first SMI sensor 500, A second SMI sensor 502 and a third SMI sensor 504). The first SMI sensor 500 , the second SMI sensor 502 , and the third SMI sensor 504 can emit respective beams of light toward the eye 506 . In some embodiments, the beams of light may be directed in different directions, which may or may not be orthogonal directions. When the beams of light are directed in different directions, a processor receiving the SMI signals generated by the SMI sensors 500, 502, and 504 can track the movement of the eye 506 in three dimensions ( e.g. , the processor can track the angular velocities of the eye 506 in three dimensions). The beams of light may impinge on the eye 506 at the same or different locations. As an example, beams of light are shown impinging on the eye 506 at the same location.

6 is a second alternative set of SMI sensors 600, 602, 604 that can be mounted to a head-mountable frame, such as one of the head-mountable frames described with reference to FIGS. 1-3B. An exemplary front view is shown. The set of SMI sensors 600, 602, 604 may form part of an optical sensor subsystem, such as one of the optical subsystems described with reference to FIGS. 1-3B or elsewhere herein. A set of SMI sensors 600, 602, 604 can be configured to emit light toward the eye.

The set of SMI sensors 600, 602, and 604 shown in FIG. 6 includes three SMI sensors ( e.g. , first SMI sensor 600, second SMI sensor 602, and third SMI sensor 604). includes The first SMI sensor 600 , the second SMI sensor 602 , and the third SMI sensor 604 can emit respective beams of light toward the eye 606 . In some embodiments, the beams of light may be directed in different directions, which may or may not be orthogonal directions. If the beams of light are directed in different directions, a processor receiving the SMI signals generated by the SMI sensors 600, 602, and 604 can track the movement of the eye 606 in three dimensions ( e.g. , the processor can track the angular velocities of the eye 606 in three dimensions). In contrast to the set of SMI sensors described with reference to FIG. 5 , two of the beams of light impinge on the eye 606 at the same location ( e.g. , the first location) and one of the beams of light strikes the eye 606 at a different location ( e.g. , the second location). It collides with the eye 606 at a location (different from the first location). Alternatively, all beams of light may be oriented to impinge eye 606 at the same location, or all beams of light may be oriented to impinge eye 606 at different locations.

The optical sensor subsystem that includes the set of SMI sensors 600 , 602 , and 604 may also include a camera 608 . Similar to the set of SMI sensors 600, 602, 604, camera 608 may be mounted on a head-mountable frame. Camera 608 may be positioned and/or oriented to acquire images of eye 606 . The images may be images of part or all of the eye 606 . A processor configured to operate the optical sensor subsystem uses images acquired by camera 608 and SMI signals generated by a set of SMI sensors 600, 602, 604 to determine rotational movement of eye 606. can be configured to track For example, the processor may use camera 608 to acquire a set of images of the eye at a first frequency. The processor may also sample the set of one or more SMI signals at a second frequency higher than the first frequency. Images and SMI signal samples may be acquired/sampled in a time-overlapping manner, or in parallel at different times. In some cases, a processor obtains one or more images; analyze the image(s) to determine the position of the eye 606 relative to the SMI sensors 600, 602, 604 and/or the beams of light emitted by the SMI sensors 600, 602, 604; If necessary, adjustments can be made to the optical sensor subsystem to ensure that the beams of light impinge on the desired structure of the eye 606 . Adjustment to the optical sensor subsystem may, for example, adjust a beam steering component to steer one or more beams of light, and address and force a specific subset of SMI sensors 600, 602, 604 to emit light. and the like ( eg , see descriptions of FIGS. 7 and 8B). Alternatively ( e.g. , alternative to relying on camera 608), SMI sensors 600, 602, 604 can be used to perform range measurements as the user moves their eyes 606 and , the range measurement can be mapped to the eye model to determine if the SMI sensors 600, 602, 604 are focused on the desired eye structure. The eye model can be a generic eye model, or an eye model created for a specific user when the optical sensor subsystem operates in training and eye model creation modes.

The optical sensor subsystem comprising the set of SMI sensors 600, 602, 604 may also illuminate the eye 606 for purposes of taking images of the eye 606 (or other purposes). Light emitters 610 and 612 may be included. Light emitters 610 and 612 may take the form of LEDs, lasers, or other light emitting elements of the display. Light emitters 610 and 612 may emit visible or non-visible light ( eg , IR light), depending on the type(s) of light that camera 608 is configured to sense. Light emitters 610 and 612 may be used to provide flood, scan, or spot illumination.

In some embodiments, the processor may determine or track the gaze vector of eye 606 using SMI signals generated by a set of SMI sensors 600 , 602 , 604 . In some embodiments, the processor may determine or track a line of sight vector using one or more images obtained using camera 608 . In some embodiments, the processor may use one or more images obtained using camera 608 in combination with the SMI signals to track the gaze vector. For example, one or more images may be used to determine the gaze vector, and the SMI signals may then be used to track movement of the eye 606 and update the gaze vector (or in other words, determine the movement of the gaze vector). can

7 is a third alternative set of SMI sensors 700, 702, 704 that can be mounted to a head-mountable frame, such as one of the head-mountable frames described with reference to FIGS. 1-3B. An exemplary side view is shown. A set of SMI sensors 700, 702, 704 may form part of an optical sensor subsystem, such as one of the optical subsystems described with reference to FIGS. 1-3B or elsewhere herein. A set of SMI sensors 700 , 702 , 704 may be configured to emit light towards eye 706 .

The set of SMI sensors 700, 702, and 704 shown in FIG. 7 includes three SMI sensors ( e.g. , first SMI sensor 700, second SMI sensor 702, and third SMI sensor 704). includes The first SMI sensor 700, the second SMI sensor 702, and the third SMI sensor 704 are sets of SMI sensors in which beams of light are described with reference to FIGS. 4A and 4B (or FIG. 5 or 6). may emit respective beams of light toward the eye 706, similar to how it may be emitted by . However, not all SMI sensors 700, 702, 704 can emit beams of light simultaneously. For example, the second and third SMI sensors 702, 704 can be part of an array of addressable SMI sensors, which array can include more than two SMI sensors in some cases. The array of SMI sensors can be coupled to circuitry that can be used to address different SMI sensors or subsets of different SMI sensors within the array of SMI sensors.

In some cases, the beams of light emitted by the second and third SMI sensors 702, 704 (and in some cases, the beams of light emitted by other SMI sensors in the array of SMI sensors) are a shared lens , may be directed towards the lens train, lens array, or one or more other optical components 708.

In some cases, the optical sensor subsystem that includes the set of SMI sensors may also include a camera 710 , which may be used similarly to the camera described with reference to FIG. 6 .

In some cases, the processor can selectively operate ( eg , activate and deactivate) different SMI sensors 702, 704 (or subsets of different SMI sensors) in the array of SMI sensors for different purposes. For example, a processor may operate (or use) circuitry to activate a particular SMI sensor, or a particular subset of SMI sensors, to have a particular focus or focal points. In some cases, the processor may cause camera 710 to acquire one or more images of eye 706 . The processor may then analyze the image(s) to determine the gaze vector of the eye 706, which is centered on a particular structure or region (or structures or regions) of the eye 706 (in the array of SMI sensors). One or more SMI sensors 702, 704 may be activated. The processor may also activate other SMI sensors, such as SMI sensor 700.

8A shows an exemplary use of the SMI sensor 800 in combination with a beam splitter 802. The SMI sensor 800 can be any of the SMI sensors described with reference to FIGS. 1-7 or any of the SMI sensors described below. A beam splitter 802 may be positioned to split the beam of light 804 emitted by the SMI sensor. The beam may be split into multiple beams 806, 808, 810 ( eg , two, three, or more beams). In some cases, beam splitter 802 can be associated with one or more lenses or beam steering components that redirect or steer multiple beams 806, 808, 810. In some cases, multiple beams 806, 808, 810 can be redirected towards a shared focal point on (or within) the eye.

8B shows an example use of the SMI sensor 850 in combination with a beam steering component 852. The SMI sensor 850 can be any of the SMI sensors described with reference to FIGS. 1-7 or any of the SMI sensors described below. Beam steering component 852 can be positioned to condition the beam of light 854 emitted by SMI sensor 850 . The processor can be configured to operate beam steering component 852 and steer the beam of light 854 to different structures or regions of the eye. In some embodiments, beam steering component 852 can include a beam focusing component or lens positioning mechanism, which can be adjusted to change the focus of a beam along its axis. . In some embodiments, beam steering component 852 may be replaced with a beam focusing component.

9A shows a first exemplary integration of the SMI sensor 900 with the display subsystem. The SMI sensor 900 may be any of the SMI sensors described with reference to FIGS. 1-7, or any of the SMI sensors described below. The display subsystem may be the display subsystem described with reference to FIGS. 1 , 3A, or 3B in some examples.

The display subsystem may include an array of display pixels 902 , 904 , 906 mounted or formed on a substrate 908 . As an example, the display subsystem is shown to include a blue pixel 902, a green pixel 904, and a red pixel 906, but the display subsystem is multiple instances of each blue pixel, green pixel, and red pixel. may include In some cases, display pixels 902, 904, 906 may include LEDs or other types of light emitting elements.

SMI sensor 900 may be mounted on substrate 908 . As an example, SMI sensor 900 is shown mounted on a substrate 908 adjacent to an array of display pixels 902 , 904 , 906 . Alternatively, SMI sensor 900 may be mounted on substrate 908 within an array of display pixels 902 , 904 , 906 ( ie , between display pixels). In some embodiments, more than one SMI sensor 900 may be mounted on the substrate 908, with each SMI sensor 900 adjacent to or adjacent to an array of display pixels 902, 904, 906. located in it In some embodiments, SMI sensor 900 may emit IR light. In other embodiments, SMI sensor 900 may emit visible light, ultraviolet light, or other types of light. In some embodiments, one or more of display pixels 902, 904, 906 may be operable as SMI sensors.

The shared waveguide 910 is a set of one or more mirrors that can be moved by a microelectromechanical system (MEMS) through the display pixels 902, 904, 906 and the beams of light emitted by the SMI sensor 900. It may be positioned to guide towards beam steering component 912 . In alternative embodiments, a set of waveguides ( eg , a set of optical fibers) directs light emitted by display pixels 902, 904, 906 and SMI sensor 900 toward beam steering component 912. can be used for A processor may operate display pixels 902, 904, 906 and beam steering component 912 to render text, numbers, or images on the display. In some cases, the display can include one or more lenses of eyeglasses, or a display positioned within a VR headset.

The shared waveguide 910 (or set of waveguides) receives the returned portion of the light emitted by the SMI sensor 900, such as the portion of light reflected or scattered by the eye, and sends the returned portion of the emitted light to the SMI sensor ( 900) into the resonant cavity.

9B shows a second exemplary integration of the SMI sensor 950 with the display subsystem. SMI sensor 950 may be any of the SMI sensors described with reference to FIGS. 1-7 or any of the SMI sensors described below. The display subsystem may be the display subsystem described with reference to FIGS. 1 , 3A, or 3B in some examples.

The display subsystem may include an array of display pixels 952 , 954 , 956 mounted or formed on a substrate 958 and an SMI sensor 950 . Display pixels 952, 954, 956 and SMI sensor 950 may be configured as described with reference to FIG. 9A.

A waveguide 960 (or set of waveguides) may be positioned to guide the beams of light emitted by the display pixels 952, 954, 956 and the SMI sensor 950 further toward the shared waveguide 962, or , or the distal portion of the shared waveguide 960 (or the distal portions of the set of waveguides) can be bent, and the light is further displaced from outcouplings of the shared waveguide 962, or from the shared waveguide 962. waveguide 962 or outcouplings of the distal portion(s) of the set of waveguides. A processor may operate display pixels 952, 954, and 956 to project text, numbers, or images onto the display.

The returning portion of the light emitted by the SMI sensor 950, such as the portion of light that is reflected or scattered from the eye, will be redirected through the waveguide(s) 962, 960 and into the resonant cavity of the SMI sensor 950. can

10 shows an example set of components 1000 that may be included in an optical sensor subsystem of an eye tracking device. The set of components 1000 is generally a subset of optical or optoelectronic components 1002, a subset of analog components 1004, a subset of digital components 1006, and a subset of system components 1008. It is divided into subsets ( eg , processors, and in some cases other control components).

A subset of optical or optoelectronic components 1002 may include a laser diode 1010 or other optical emitter with a resonant cavity. Components 1002 may also include a photodetector 1012 ( eg , a photodiode). Photodetector 1012 can be integrated in the same epitaxial stack as laser diode 1010 ( e.g. , above, below, or adjacent to laser diode 1010), or stacked with or in conjunction with laser diode 1010. It may be formed as a separate component that is positioned adjacently. Alternatively, photodetector 1012 may be replaced or supplemented with circuitry that generates an SMI signal electronically ( ie , without a photosensitive element) by measuring the match voltage or drive current of laser diode 1010. Laser diode 1010, in combination with photodetector 1012 or alternative circuitry for generating an SMI signal, may be referred to as an SMI sensor.

The subset of optical or optoelectronic components 1002 may also include module level optics 1014, and/or system level optics 1016 integrated with laser diode 1010 and/or photodetector 1012. can Module level optics 1014 and/or system level optics 1016 may include, for example, lenses, beam splitters, beam steering components, and the like. Module level optics 1014 and/or system level optics 1016 can determine where emitted light and returned light ( eg , light scattered from the eye) are directed.

A subset of the analog components 1004 can include a digital to analog converter (DAC) 1018 and a current regulator 1020 to convert the drive current to the analog domain and provide it to the laser diode 1010 . Components 1004 may also include components 1022 to ensure that laser diode 1010 is operated in compliance with operational safety specifications. Components 1004 also include a transimpedance amplifier (TIA) and/or other amplifiers 1024 for amplifying the SMI signal generated by photodetector 1012 and for converting the amplified SMI signal to the digital domain. It may include an analog-to-digital converter (ADC). Components 1004 may also include components for correcting the SMI signal as it is amplified or otherwise processed.

In some cases, a subset of analog components 1004 may interface ( eg , multiplex) with more than one subset of optical or optoelectronic components 1002 . For example, components 1004 may interface with two, three, or more SMI sensors.

A subset of digital components 1006 schedules ( e.g. , associates) providing drive current to laser diode 1010 with providing digitized photocurrent obtained from photodetector 1012 to system components 1008. It may include a scheduler 1026 for Components 1006 can also include drive current waveform generator 1028 that provides digital drive current to DAC 1018 . Components 1006 may further include a digital processing chain for processing the amplified and digitized output of photodetector 1012 . The digital processing chain includes, for example, a time-domain signal pre-conditioning circuit 1030, a fast Fourier transform (FFT) engine 1032, a frequency-domain signal pre-conditioning circuit 1034, and distance and/or speed An estimator 1036 may be included. In some cases, some or all of components 1006 may be instantiated by a set of one or more processors.

A subset of system components 1008 may include, for example, a system-level scheduler 1038, which scheduler 1038 allows an SMI sensor (or SMI sensors) or other components to detect eye position ( eg , gaze vector) or motion ( eg , angular velocity) can be scheduled. Components 1008 may also include other sensors, such as a camera 1040 or an inertial measurement unit (IMU) 1042 . In some cases, components 1008 (or their processor) may include one or more SMI signals obtained from one or more subsets of components 1002, one or more images obtained from camera 1040, and /or other measurements ( eg , measurements obtained by IMU 1042) may be used to track eye position or movement. Components 1008 may also include sensor fusion applications 1044 and various other applications 1046 .

Laser diode 1010 can be driven using various types of drive currents. For example, laser diode 1010 may be driven with DC current ( eg , in a DC drive mode) for the purpose of performing Doppler analysis on the SMI signal generated by photodetector 1012 . Alternatively, the laser diode 1010 may be driven in a frequency modulated continuous wave (FMCW) mode ( eg , with a triangular wave drive current) when performing ranging. Alternatively, laser diode 1010 can be driven in a harmonic drive mode ( eg , with an IQ-modulated drive current) when determining the relative displacement of the eye.

11A shows a first example method 1100 for tracking eye movement using a set of one or more SMI sensors 1104 . The method 1100 includes operating an optical sensor subsystem 1102 to cause a set of one or more SMI sensors 1104 to emit a set of one or more beams of light toward an eye of a user. Optical sensor subsystem 1102 and SMI sensor(s) 1104 may be configured similarly to any of the optical sensor subsystems and SMI sensors described herein.

At 1106 , method 1100 may include receiving a set of one or more SMI signals from a set of one or more SMI sensors 1104 .

At 1108 , method 1100 may include tracking rotational movement of the eye using the set of one or more SMI signals. The action(s) at 1108 may include (at 1110) estimating gaze and angular velocity of the eye using, for example, the SMI signals and Doppler interferometry. The operation(s) at 1108 may also include estimating (at 1112) a range (or distance) to the eye, or (at 1114) estimating a surface quality ( eg , surface texture) of the eye. The estimated range(s) or surface quality(s) are not only used to estimate rotational movements of the eye, but also (at 1116) the position of the eye, or the position of the eye on which the set of one or more SMI sensors 1104 are focused. can be used to determine the structure(s).

At 1118 , method 1100 may include determining 1108 eye movement using the output(s) of the action(s).

At 1120, method 1100 uses the output(s) of action(s) 1108 to perform a gaze wakeup event ( e.g. , the user opens their eyes, or the user looks in a certain direction, or the user performing a series of eye movements). In some cases, the action(s) of 1120, or other actions, may be a gaze slip event ( eg , the user closes their eyes, the user looks in a particular direction, or the user performs a particular series of eye movements), a blink identifying the event, or other events.

At 1122, method 1100 includes performing an action ( eg , powering on or off a head mounted display or other device, answering a call, activating an application, adjusting volume, etc.) in response to the identified event of a particular type. can include

At 1124 , method 1100 may include performing Doppler odometry to determine a change in position of the eye gaze vector. In some cases, Doppler odometry may be performed using an extended Kalman filter (EKF).

At 1126, the method 1100 includes updating a gaze-to-head mounted display (gaze-HMD) vector ( ie , determining how the eye gaze vector intersects the display, or how the eye gaze vector has moved relative to the display). can include

At 1128 , method 1100 may include causing a display subsystem for the HMD (or other display) to adjust rendering of text, numbers, or images on the display. In some cases, accommodation may be responsive to classifying eye movement ( eg , as euphemism, saccade, gaze, nystagmus, or blink).

FIG. 11B shows a set of one or more SMI sensors 1104 and a camera 1152 or other sensors ( e.g. , IMU 1154, outward-facing camera (OFC)) using the operations described in FIG. 1156), etc.) shows a second example method 1150 for tracking eye movement. Camera 1152 may be configured similarly to other cameras described herein.

At 1158 , method 1150 may include acquiring a set of one or more images of the eye using camera 1152 . In some embodiments, camera 1152 may acquire a set of images at a first frequency, and the SMI signal(s) generated by SMI sensor(s) 1104 may acquire a second frequency ( e.g. , first a second frequency synchronized with the frequency). The frequencies may be the same or different, but in some embodiments the second frequency may be greater than the first frequency. In this way, camera 1152, which typically consumes more power and produces greater amounts of data, determines eye position or produces gaze vector data at a lower frequency, and the SMI sensor(s) properly can be used to ensure focus and produce good data; The SMI sensor(s) 1104, which typically consume less power, provide more or less continuous, and at higher frequencies, eye movement (or gaze vectors) between image captures by the camera 1152. can be used to track In some embodiments, images acquired by camera 1152 can be used to create, update, or customize an eye model, which can detect the beam(s) of light emitted by the SMI sensor(s). It can be used to direct or focus.

At 1160 , method 1150 may include estimating a gaze vector (or position) of the eye based on the image(s) acquired by camera 1152 .

At 1162 , method 1100 may include determining or updating a head-HMD vector (head-HMD vector). In other words, the method 1100 can determine how the user's head is positioned relative to the display.

At 1164 , method 1100 may include performing visual-Doppler odometry to determine a change in position of the eye gaze vector. In some cases, visual-Doppler odometry may be performed using an extended Kalman filter (EKF). In contrast to the Doppler odometry performed in method 1100, the visual-Doppler odometry performed in 1164 uses an SMI-based motion (or position) analysis of the eye as an image-based position (or gaze vector) analysis of the eye. can be fused with

At 1126, the method 1100 includes updating a gaze-to-head mounted display (gaze-HMD) vector ( ie , determining how the eye gaze vector intersects the display, or how the eye gaze vector has moved relative to the display). can include

At 1166, method 1100 optionally uses the output of IMU 1154 or look-ahead camera 1156 ( i.e. , a camera focused on the environment around the user relative to the user's eyes) to perform inertial odometry, video It can perform odometry, or video-inertial odometry. Video-inertial odometry may then be used to determine or update the HMD-World (HMD-World) vector at 1168 .

At 1170 , method 1100 may include determining or updating a gaze-world vector. Such vectors may be useful, for example, to augment the user's reality through glasses.

At 1128, the method 1100 may include causing a display subsystem for the HMD (or other display) to adjust rendering of text, numbers, or images on the display ( e.g. , in an AR or VR environment). . In some cases, accommodation may be responsive to classifying eye movement ( eg , as euphemism, saccade, gaze, nystagmus, or blink).

12A and 12B show how one or more surfaces or structures of eye 1200 may be mapped using one or more sets of SMI sensors. As an example, FIGS. 12A and 12B show a single SMI sensor 1202 . In some examples, SMI sensor 1202 can be replaced with multiple SMI sensors ( eg , multiple individual SMI sensors or an array of SMI sensors). The SMI sensor(s) may be any of the SMI sensors described with reference to FIGS. 1-11B.

12A and 12B each show two side views of the same eye 1200 . The first side view in each figure ( i.e. , side views 1204 and 1206) represents a cross-section of eye 1200, and the second side view in each figure ( i.e. , side views 1208 and 1210) represents the SMI It represents a computer-generated model of various eye structures identified by a processor after analyzing the SMI signal generated by the sensor or multiple SMI signals generated by multiple different SMI sensors. 12A shows the eye 1200 in a first position, and FIG. 12B shows the eye 1200 in a second position.

In the case of a single SMI sensor 1202, the SMI sensor 1202 is head by a MEMS or other structure 1212 that causes the beam of light 1214 emitted by the SMI sensor 1202 to be scanned across the eye 1200. A set of one or more optical elements that can be mounted on a mountable frame or steered such that the beam of light 1214 emitted by the SMI sensor 1202 scans the beam of light 1214 across the eye 1200. can be received by Alternatively, the beam of light 1214 can be split using a beam splitter, and multiple beams of light can impinge on the eye 1200 either simultaneously or sequentially. Alternatively, the SMI sensor 1202 can be mounted to a head-mountable frame in a fixed location and the user can move their eyes 1200 to different positions while the SMI sensor 1202 emits a beam of light. may be requested.

A processor, such as any of the processors described herein, receives SMI signal(s) generated by a set of one or more SMI sensors, and uses the SMI signal(s) to locate a set of points on or within an eye. A set of ranges can be determined for Ranges can include absolute ranges or relative ranges. The processor may then use the set of ranges to create a map of the at least one structure of the eye. The map may be a two-dimensional (2D) map or a three-dimensional (3D) map.

In some embodiments, the processor may be configured to identify, using the map, a structure of the eye or a boundary between a first structure of the eye and a second structure of the eye. The identified structure(s) may include, for example, one or more of iris 1218, sclera 1220, pupil 1222, lens 1224, limbus 1226, eyelid, and the like.

In some embodiments, the processor may operate an optical sensor subsystem ( eg , a MEMS, one or more optical elements, a beam splitter, etc.) to direct one or more beams of light toward the identified structure of the eye. In some cases, the structure may be more diffuse than other structures of the eye.

In some embodiments, the processor may be configured to determine the eye's gaze vector 1216 using the map. In some embodiments, the processor may also or alternatively be configured to obtain or construct a Doppler cloud using a set of one or more SMI signals. The one or more SMI signals correspond to simultaneously or sequentially projecting or emitting multiple beams of the set of one or more beams and/or scanning at least one beam of the set of one or more beams. A Doppler cloud may be obtained or constructed with or without VCSEL wavelength modulation. The processor may also or alternatively obtain or construct a depth cloud. A depth cloud may be obtained or constructed using only VCSEL wavelength modulation.

For example, when the wavelength(s) of the beam(s) of light emitted by one or more SMI sensors is modulated, and the beam(s) of light emitted by at least one SMI sensor is scanned and/or multiple beams are emitted. When done, a Doppler cloud, and/or a depth cloud may be obtained or constructed. Additionally or alternatively, a Doppler cloud may be obtained or constructed when the wavelength(s) of the beam(s) of light emitted by one or more SMI sensors are not modulated. As mentioned above, a single or multiple frames of a Doppler cloud can be considered a differential depth cloud. Using measurements of single or multiple frames of the Doppler cloud processed in real time, a predefined and/or locally calibrated difference map or library may be matched and/or eye tracking or location information may be extracted. As described herein, locally calibrated difference maps or libraries may be obtained by means including, but not limited to, cameras, depth clouds, and the like. In addition, Doppler clouds, alone or fused with depth clouds or other sensing modalities ( eg , eye camera images, motion sensors, etc.), can be used to provide an accurate and efficient way to track eye movement or positional information.

The foregoing description, for purposes of explanation, uses specific nomenclature to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art after reading this description that the specific details are not essential to practicing the described embodiments. Accordingly, the foregoing descriptions of specific embodiments described herein have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. It will be apparent to those skilled in the art after reading this description that many modifications and variations are possible in light of the above teachings.

As noted above, one aspect of the present technology may be the collection and use of data available from a variety of sources, including biometric data (eg, the surface quality of a user's skin or fingerprint). This disclosure contemplates that, in some examples, this collected data may include personal information data that uniquely identifies a particular person or that can be used to identify, locate, or contact him. Such personal information data may include, for example, biometric data (eg, fingerprint data) and data associated therewith (eg, demographic data, location-based data, phone numbers, email addresses, home addresses, a user's health or fitness data or records relating to levels (eg, vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information).

The present invention recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, personal information data may be used to authenticate users to access their devices or to collect performance metrics for users' interactions with augmented or virtual worlds. Additionally, other uses for personal information data that benefit the user are also contemplated by the present invention. For example, health and fitness data can be used to provide insight into a user's general wellness, or as positive feedback to individuals using technology to pursue wellness goals.

The present invention contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will adhere to well-established privacy policies and/or privacy practices. In particular, such entities must implement and consistently use privacy policies and practices that are generally recognized to meet or exceed industrial or administrative requirements for keeping personal information data private and secure. Such policies should be easily accessible by users and should be updated as the collection and/or use of data changes. Personal information from users should be collected for legitimate and appropriate uses of the entity, and should not be shared or sold outside of these legitimate uses. Further, such collection/sharing must occur after receiving the notified consent of the users. Additionally, such entities are committed to taking any necessary steps to protect and secure access to such personal information data and to ensure that others having access to such personal information data adhere to their privacy policies and procedures. should be considered Additionally, such entities may themselves be evaluated by third parties to demonstrate their adherence to widely accepted privacy policies and practices. Additionally, policies and practices must be tailored to the specific types of personal information data collected and/or accessed and adapted to applicable laws and standards, including jurisdiction specific considerations. For example, in the United States, the collection of certain health data or access thereto may be governed by federal and/or state laws, such as the US Health Insurance Portability and Accountability Act (HIPAA); Health data from other countries may be subject to different regulations and policies and should be treated accordingly. Accordingly, different privacy practices must be maintained for different types of personal data in each country.

Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present invention contemplates that hardware and/or software components may be provided to prevent or block access to such personal information data. For example, in the case of advertisement delivery services, the present technology allows users to select "agree" or "disagree" of participation in the collection of personal information data during registration for the service or at any time thereafter. It can be configured to allow In another example, users may choose not to provide data to targeted content delivery services. In another example, users may choose to limit the length of time data is maintained or to forbid development of a default profile for the user entirely. In addition to providing "I agree" and "I do not agree" options, the present invention contemplates providing notifications regarding access or use of personal information. For example, a user may be notified upon downloading an app that their personal data will be accessed, and then reminded immediately before the personal data is accessed by the app.

Moreover, it is the intent of this disclosure that personal information data should be managed and processed in a manner that minimizes the risks of unintended or unauthorized access or use. Risk can be minimized by limiting the collection of data and deleting data when it is no longer needed. In addition, and when applicable, including as applicable to certain health-related applications, data de-identification may be used to protect user privacy. Control how data is stored, where appropriate, by removing certain identifiers (eg, date of birth, etc.); by controlling the amount or specificity of stored data (eg, collecting location data at the city level rather than at the address level); De-identification may be facilitated by (eg, by aggregating data across users), and/or by other methods.

Thus, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, this disclosure also indicates that various embodiments may also be implemented without the need to access such personal information data. also consider That is, various embodiments of the present technology are not inoperable due to the lack of all or some of such personal information data. For example, content may include content requested by a device associated with a user, other non-personal information available for content delivery services, or non-personal information data such as publicly available information or minimal amounts of personal information. can be selected and communicated to users by inferring preferences based on

Claims (27)

As an eye tracking device,
a head-mountable frame;
an optical sensor subsystem mounted to the head-mountable frame and including a set of one or more self-mixing interferometry (SMI) sensors; and
It includes a processor, the processor comprising:
operating the optical sensor subsystem to cause the set of one or more SMI sensors to emit a set of one or more beams of light toward an eye of a user;
receive a set of one or more SMI signals from the set of one or more SMI sensors;
An eye tracking device configured to track movement of the eye using the set of one or more SMI signals. The eye tracking device of claim 1 , wherein tracking the motion of the eye includes estimating an angular velocity of the eye. According to claim 1,
the set of one or more SMI sensors includes at least two SMI sensors;
wherein the tracked movement of the eye includes tracked angular velocities of the eye in two dimensions. According to claim 1,
the set of one or more SMI sensors includes at least three SMI sensors;
wherein the tracked movement of the eye includes tracked angular velocities of the eye in three dimensions. The method of claim 1, wherein the head-mountable frame comprises:
a first lens rim and a second lens rim;
a bridge connecting the first lens rim to the second lens rim;
a first temple connected to the first lens rim; and
and a second temple coupled to the second lens rim. According to claim 1,
and a display subsystem comprising a display mounted to the head-mountable frame. According to claim 6,
the processor,
classifying the movement of the eye as at least one of blinking, smooth pursuit, saccade, fixation, or nystagmus;
and cause the display subsystem to adjust rendering of one or more images on the display in response to the classified movement of the eye. According to claim 7,
the processor,
configured to quantify the classified movement of the eye, wherein:
wherein the rendering of the one or more images is further adjusted in response to the quantified movement of the eye. According to claim 6,
the processor,
and cause the display subsystem to change a state of the display in response to the movement of the eye. According to claim 6,
the display subsystem includes an array of display pixels mounted on a substrate;
wherein the set of one or more SMI sensors comprises at least one SMI sensor mounted on the substrate, adjacent to or within the array of display pixels. According to claim 1,
The optical sensor subsystem,
and a beam splitter positioned to split a beam of light emitted by an SMI sensor in the set of one or more SMI sensors. According to claim 1,
The optical sensor subsystem,
and the waveguide positioned to direct a beam of light emitted by an SMI sensor in the set of one or more SMI sensors to a plurality of outcouplings of the waveguide. According to claim 1,
The optical sensor subsystem,
a beam steering component positioned to steer a beam of light emitted by an SMI sensor in the set of one or more SMI sensors;
wherein the processor is configured to operate the beam steering component and steer the beam of light. According to claim 1,
The optical sensor subsystem,
a lens positioned to focus a beam of light emitted by an SMI sensor in the set of one or more SMI sensors; and
includes a lens positioning mechanism;
wherein the processor is configured to operate the lens positioning mechanism and focus the beam of light on the structure of the eye. According to claim 1,
The set of one or more SMI sensors,
an array of SMI sensors; and
circuitry configured to process different SMI sensors or subsets of different SMI sensors in the array of SMI sensors;
wherein the processor is configured to selectively operate the different SMI sensors or subsets of the different SMI sensors using the circuitry. According to claim 1,
the processor,
and determine a set of ranges for a set of points on or within the eye. According to claim 16,
the processor,
and generate a map of the at least one structure of the eye using the set of ranges. According to claim 17,
the processor,
using the map, identify the structure of the eye;
and operate the optical sensor subsystem to direct at least one beam of light in the set of one or more beams of light towards the identified structure of the eye. 17. The eye tracking device of claim 16, wherein the set of ranges comprises a set of absolute ranges. 17. The eye tracking device of claim 16, wherein the set of ranges comprises a set of relative ranges. As an eye tracking device,
a set of one or more self-mixing interferometry (SMI) sensors;
camera; and
It includes a processor, the processor comprising:
cause the camera to acquire, at a first frequency, a set of images of the user's eye;
cause the set of one or more SMI sensors to emit a set of one or more beams of light toward the eye of the user;
sample a set of one or more SMI signals generated by the set of one or more SMI sensors at a second frequency greater than the first frequency;
determine a gaze vector of the eye using at least one image in the set of images;
An eye tracking device configured to track movement of the eye using the set of one or more SMI signals. According to claim 21,
the processor,
and update the gaze vector using the tracked movement of the eye. As a method of tracking eye movement,
operating an optical sensor subsystem to cause a set of one or more self-mixing interferometry (SMI) sensors within the optical sensor subsystem to emit one or more sets of beams of light toward the eye of a user;
receiving a set of one or more SMI signals from the set of one or more SMI sensors; and
tracking the movement of the eye using the set of one or more SMI signals. 24. The method of claim 23, wherein tracking the movement of the eye using the set of one or more SMI signals comprises:
estimating the angular velocity of the eye using the set of one or more SMI signals. According to claim 23,
simultaneously projecting multiple beams of the set of one or more beams; or
sequentially emitting the plurality of beams; or
further comprising at least one of scanning at least one beam of the set of one or more beams, wherein:
Tracking the movement of the eye using the set of one or more SMI signals comprises:
constructing a Doppler cloud using the set of one or more SMI signals. 26. The method of claim 25, wherein tracking the movement of the eye using the set of one or more SMI signals comprises:
and extracting eye position information by matching one or more frames of the Doppler cloud to a differential map. According to claim 23,
simultaneously projecting multiple beams of the set of one or more beams; or
sequentially emitting the plurality of beams; or
modulating the emitted set of one or more beams during the step of scanning at least one beam of the set of one or more beams;
here,
Tracking the movement of the eye using the set of one or more SMI signals comprises:
constructing a depth cloud using the set of one or more SMI signals.

Images