CN115543073A

CN115543073A - Eye tracking data filtering

Info

Publication number: CN115543073A
Application number: CN202210766895.4A
Authority: CN
Inventors: J·威尔逊; T·埃布莱; S·韦贝尔; H·普图库蒂尔拉贾戈帕尔; A·加佩尔; R·G·肖拉普尔
Original assignee: Apple Inc
Current assignee: Apple Inc
Priority date: 2021-06-30
Filing date: 2022-06-30
Publication date: 2022-12-30
Also published as: GB2608705B; GB2608705A; JP2023008928A; DE102022116074A1; GB202209217D0

Abstract

The present disclosure relates to eye tracking data filtering. Eye tracking is performed by: determining an initial pupil position of a user relative to a lens located in front of the user; detecting a change in pupil position relative to the lens to an updated pupil position relative to the lens; and determining that the updated pupil location is outside of a bounding box associated with the lens. The updated pupil position is a replacement pupil position having a replacement pupil position within a bounding box associated with the lens, and the updated pupil position is used for eye tracking functions. Eye tracking is also performed by: determining that a first pixel associated with a gaze direction is outside a visibility region; identifying replacement pixels within the visible region; determining an updated gaze angle based on the replacement pixels; and performing eye tracking using the updated gaze angle.

Description

Eye tracking data filtering

Background

The present disclosure relates generally to image processing. More particularly, but not by way of limitation, the present disclosure relates to techniques and systems for generating and managing eye tracking data for improved eye tracking techniques.

Eye tracking is a technique employed in many fields such as gaze detection, pose estimation, face analysis and recognition. Eye tracking generally forms the basis of these operations and can be thought of as a process of electronically locating a person's point of view or tracking and recording the movement of a person's point of view. In practice, eye tracking is provided by locating and tracking pupil position and gaze direction. However, the sensors used to track the eye are often shifted or dithered so that the eye tracking data is not always perfectly calibrated. Thus, the eye tracking data may be jittery or invalid. There is a need for an improved technique for managing eye tracking data.

Drawings

Fig. 1A and 1B show exemplary diagrams of a setup for eye tracking and performing pupil localization.

Fig. 2 illustrates a flow diagram of a technique for managing pupil location information for an eye tracking technique, according to one or more embodiments.

Fig. 3 illustrates a flow diagram of a technique for optimizing pupil position for eye tracking according to one or more embodiments.

Fig. 4A and 4B illustrate example diagrams of gaze direction relative to visibility regions in accordance with one or more embodiments.

Fig. 5A and 5B illustrate a flow diagram of a technique for optimizing gaze angles for eye tracking techniques, according to one or more embodiments.

Fig. 6 illustrates a flow diagram of a technique for identifying replacement pixels to optimize a gaze angle in accordance with one or more embodiments.

FIG. 7 illustrates, in block diagram form, a multi-function electronic device in accordance with one or more embodiments.

FIG. 8 illustrates, in block diagram form, a computer system in accordance with one or more embodiments.

Detailed Description

The present disclosure relates to systems, methods, and computer-readable media for optimizing and smoothing eye tracking data to enhance the performance of eye tracking technologies.

In one or more embodiments, the pupil location is held within a predetermined area. The lens through which the pupil is monitored may have a predetermined area that is a known calibration area, while other areas of the lens may be less calibrated. Thus, the pupil position should be maintained within the calibration zone. In one or more embodiments, an initial pupil location is determined relative to a lens located in front of the eye. A change in pupil position relative to the lens to an updated pupil position relative to the lens may be detected. If the updated pupil location is outside of the determined bounding box (e.g., the bounding box associated with the calibration area), an alternate pupil location within the bounding box is selected. The updated pupil position is then used in the eye tracking technique. In some implementations, to avoid jitter, the eye tracking system may change the pupil position in a series of frames such that the pupil position transitions from an initial pupil position to an alternate pupil position, e.g., based on a time-based bradykinetic function.

In one or more embodiments, the gaze direction may be optimized for use in eye tracking techniques. Specifically, the gaze direction is optimized such that the user is gazing at the visibility region. The first pixel may be determined to be associated with a gaze direction outside of the visibility region. Replacement pixels within the visible region are identified. The replacement pixel may be selected along a vector from the center of the field of view to the first pixel. A gaze angle is determined based on the location of the replacement pixels and used to perform an eye tracking function.

For the purposes of this disclosure, the term "gaze origin" refers to the center of the eye at which the gaze is determined.

For the purposes of this disclosure, the term "pupil location" refers to the location on the surface of the eye where the pupil is located.

For the purposes of this disclosure, the term "gaze direction" is the direction of gaze originating from the origin of gaze and passing through the pupil location.

In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed concepts. Some of the figures of the present disclosure represent structures and devices in block diagram form as part of this description to avoid obscuring the novel aspects of the disclosed concepts. In the interest of clarity, not all features of an actual implementation may be described. Additionally, as part of this specification, some of the figures of the present disclosure may be provided in the form of flow charts. The blocks in any particular flow diagram may be presented in a particular order. However, it should be understood that the particular order of any given flow diagram is merely to illustrate one embodiment. In other embodiments, any of the various elements depicted in the flowcharts may be deleted, or the illustrated sequence of operations may be performed in a different order, or even simultaneously. Further, other embodiments may include additional steps not shown as part of the flow diagram. Moreover, the language used in the present disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. Reference in the present disclosure to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to "one embodiment" or "an embodiment" should not be understood as necessarily all referring to the same embodiment.

It will be appreciated that in the development of any such actual implementation, as in any software and/or hardware development project, numerous decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. It will also be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art of designing and implementing graphical modeling systems having the benefit of this disclosure.

Referring to fig. 1A, an exemplary view of an eye 100 is presented. For purposes of this disclosure, eye 100 includes a gaze origin 105, which refers to the center of the eye in 3D space. The eye 100 also includes a pupil location 110. Pupil location 110 refers to the location on the surface of eye 100 where the pupil is located. In one or more embodiments, pupil location 110 may refer to the central location of the pupil on the surface of the eye. Gaze origin 105 and pupil location 110 may be used to determine gaze direction 115. Gaze direction 115 may refer to a gaze direction originating from gaze origin 105 and passing through pupil location 110. In one or more embodiments, the gaze direction 115 may be defined as a vector generated from the gaze origin 105 and passing through the pupil location 110.

Turning to fig. 1B, this figure is an exemplary diagram of an eye 100 for which eye tracking is being performed. Eye tracking may be performed by a device 125 that includes a lens 120. When the eye 100 views a subject through the lens 120, only some of the pupil data is valid based on calibration of the system. The pupil location 110 may affect the calibration of the system. For example, hardware limitations may result in only a portion of the lens 120 being calibrated, although the pupil location 110 may still reach an uncalibrated portion. Thus, pupil boundary 130 may define a portion of the pupil data in lens 120 that is determined to be valid. In some embodiments, the intersection of pupil boundary 130 and lens 120 may identify a bounding box on the lens.

Fig. 2 illustrates a flow diagram of a technique for managing pupil location information for an eye tracking technique, according to one or more embodiments. In particular, fig. 2 depicts a technique for clamping the pupil location within the bounding box of the lens. By clipping the pupil location to a portion of the lens determined to provide valid pupil data, the system can avoid having to extrapolate data from an uncalibrated portion of the lens, causing the image to appear more distorted than within the bounding box. For example, if the display device is moved or displaced in front of the eyes, the pupil position may shift frequently, causing jitter to appear on the screen. While the various processes shown in fig. 2 are shown in a particular order, it should be understood that the various processes described may be performed in a different order. Further, not all of the various processes need necessarily be performed.

Flowchart 200 begins at 205 where an initial pupil position of a user relative to a lens located in front of the user is determined. For example, the lens may be positioned in front of the user's eye for which the initial pupil position is to be determined. The flow diagram 200 continues at 210 where a change in pupil position relative to the lens is detected. As a result, an updated pupil position relative to the lens is determined.

At block 215, the updated pupil position is compared to a bounding box associated with the lens. As described above, the bounding box may refer to a portion of the lens determined to provide valid pupil data. The bounding box may be determined, for example, by the intersection of the effective pupil boundary with the lens, as described above with reference to fig. 1B.

The flow chart continues at 220 where it is determined whether the updated pupil position is outside the bounding box. If it is determined at 220 that the updated pupil location is not outside the bounding box, the flow chart proceeds to 225 and the updated pupil location is used as is for the eye tracking function.

Returning to 220, if the pupil location is determined to be outside of the bounding box, the flow diagram continues to block 230. At block 230, alternate pupil locations within the bounding box are identified. The replacement pupil location may be determined in a variety of ways. For example, the nearest pupil location within the bounding box may be used. For example, locations on the boundaries of the bounding box may be used. The flow chart then ends at block 235 where the replacement pupil location is used for the eye tracking function. In particular, in some embodiments, the replacement pupil location is used to calculate a warping function to render an image on a screen, as described above with reference to fig. 1B.

In some implementations, utilizing alternate pupil locations may cause the data to jump as the eye moves. For example, data jumps may result in distorted jumps. In some implementations, the distortion jump can be avoided by applying a bradykinetic function to the pupil position. Fig. 3 illustrates a flow diagram of a technique for optimizing pupil position for eye tracking according to one or more embodiments, e.g., as described above for block 230 of fig. 2. While the various processes shown in fig. 3 are shown in a particular order, it should be understood that the various processes described may be performed in a different order. Further, not all of the various processes need necessarily be performed.

The flow diagram 300 begins at 305 where alternate pupil locations within a bounding box are identified. In accordance with one or more embodiments, an alternate pupil location may be selected based on the closest location within the bounding box to the actual pupil location. In some embodiments, the replacement pupil location may be selected as a location on the boundary of the bounding box.

The flow diagram continues at block 310 where a bradykinetic function is applied to the pupil position based on the actual pupil position and the replacement pupil position. In some embodiments, the slow-motion function may be a time-based function. Thus, at block 315, rather than simply replacing the actual pupil location with a replacement pupil location, a set of intermediate pupil locations may be determined based on a bradykinetic function. According to one or more embodiments, the inching function can be a linear function, a cubic inching function, or the like. In this way, the set of intermediate pupil positions may slow the pupil position of the user to an alternate pupil position.

The flow diagram 300 continues at block 320 where the set of intermediate pupil positions is applied for a series of frames. For example, the set of intermediate pupil positions may include a predetermined number of intermediate pupil positions, which may correspond to the number of frames to which the intermediate pupil positions apply. Alternatively, in some embodiments, the number of intermediate pupil positions may be determined by the difference in actual and replacement pupil positions. For example, if the difference in linearity between the actual pupil position and the replacement pupil position is small, fewer frames may be needed, and thus fewer intermediate values may be needed to reach the replacement pixel position. Conversely, if the linear difference between the actual pupil position and the replacement pupil position is large, more frames, and therefore more intermediate values, may be needed to reach the replacement pupil position in order to effectively slow the user. The set of intermediate pupil positions is applied such that a subsequent series of frames presented to the user may utilize a series of intermediate pupil positions for the series of frames.

The flow diagram 300 ends at block 325, where the replacement pupil location is used for the next frame. That is, after the set of intermediate pupil positions is used for a series of frames to be rendered based on the replacement pupil position rather than the actual pupil position. In some implementations, the replacement pupil location can then be used to render content for presentation to the user until a new pupil location is detected, at which point the process described in fig. 2 begins again.

According to some embodiments, other problems may occur, for example, based on the user's gaze. For example, the display may have an area associated with an area that includes valid pixel information. Thus, the gaze direction may be optimized to ensure that the user is gazing at pixels within the effective visibility region. Referring to FIG. 4A, a simplified block diagram of a screen, such as screen 425, is presented. The screen 425 may be any type of display located in front of the user, such as a display in a mobile device (such as a tablet or mobile phone) or a head mounted display device. A portion of the display may be visible and is therefore referred to as a visibility region. In some embodiments, the visibility region is a portion of the screen that is visible through the lens 430. Thus, the lens cannot "see" every part of the display. In addition, not all of the screen 425 visible through the lens 430 may include valid data. For example, the outer portion of lens 430 may be difficult to calibrate or may not otherwise be calibrated. As such, the screen 425 may include a visibility area 435 that is determined to contain valid pixel data. Additionally or alternatively, the determination of the visible region may be based on what the hardware, the technical specification of the eye tracking system, or the software defined region can see.

Turning to FIG. 4B, an example view of a screen from an eye is depicted. According to some embodiments, the visibility region 435 may be defined by a visibility mask based on a portion of the screen determined to provide valid pixel data. In this way, if the user's gaze is directed outside of the visibility region 435, an alternate gaze location may be utilized. As shown, the user's gaze 460 is directed outside of the visibility region 435 but within the screen 440. The screen 440 may be a screen of a mobile device, such as a tablet, mobile phone, wearable device, etc., located in front of the user. To provide valid data, an alternate gaze location may be selected from within the visibility region. The replacement pixel may be selected, for example, based on the nearest pixel within the visibility region 435. In some implementations, the replacement pixels can be selected by identifying the center of the field of view 450. The center of the field of view 450 may be based on the position of the eyes relative to the screen. The center of the field of view 450 may be based on the position of the center of the user's eyes relative to the screen. In some implementations, the replacement pixel 455 for the gaze target 460 may be selected such that the replacement pixel 455 is within the visible region. The replacement pixel may be determined using a vector 470 along the target location originating from the center of the field of view 450 and pointing at the gaze 460. According to one or more implementations, replacement pixel 455 may then be used for eye tracking techniques. Furthermore, in some implementations, the eye tracking technique may also rely on gaze angles that may be determined based on the replacement pixels 455.

Fig. 5A illustrates a flow diagram of a technique for managing gaze information for an eye tracking technique, according to one or more embodiments. In particular, fig. 5A depicts a technique for recalculating the gaze direction of a user in order to utilize valid pixel data. For clarity, the description of fig. 3 is described with reference to fig. 1 and 4 as described above. While the various processes shown in fig. 5 are shown in a particular order, it should be understood that the various processes described may be performed in a different order. Further, not all of the various processes need necessarily be performed.

The flow diagram 500 begins at block 505, where a pixel location is detected as being associated with a gaze direction. For example, the pixel location may be associated with a target location at which the user gazes. The pixel locations may be determined relative to a screen positioned in front of the user's eye 100. At block 510, it is determined whether the detected pixel location is outside of the visibility region. As described above with reference to fig. 4B, the visibility region 435 can be a subset of the screen 425 that includes valid pixel data. The effective pixel data may be based on a subset of the lenses 430 through which the screen 425 is visible.

If it is determined at block 510 that the pixel location is not outside the visibility region, the flow diagram 500 continues to block 515. At block 515, a gaze angle value is determined based on the detected pixel locations. The gaze angle may describe the angle between the user's eyes and the screen according to the determined pixel locations. The flow chart ends at block 520 where the determined gaze angle and pixel location are used to process the next frame.

Returning to block 510 of flowchart 500, if it is determined that the detected pixel location is outside the visibility region, flowchart 500 continues at block 525. At block 525, alternate pixel locations within the visibility region are determined. In one or more implementations, the closest pixel to the detected pixel, e.g., within the visibility region, may be selected as the replacement pixel. Furthermore, in some implementations, the alternate pixel location may be selected based on the center point of the field of view, as will be described in detail below with reference to fig. 6.

The flow diagram 500 continues at block 530 where the gaze angle is determined based on the location of the replacement pixel. The gaze angle may describe the angle between the user's eyes and the screen according to the determined alternate pixel locations. The flow chart ends at block 535 where the determined gaze angle and replacement pixel location are used to process the next frame.

According to some embodiments, the client and compositor may render the frames in a two-stage process using eye tracking data. In some embodiments, eye tracking data may be communicated between the client and the synthesizer, and/or historical eye tracking data may be used to improve visual quality and/or meet hardware limitations. In particular, consistency between the client and the synthesizer may be improved by sharing at least some eye tracking parameters between the two. Fig. 5B illustrates a flow diagram of a technique for managing eye tracking data according to one or more embodiments. In particular, fig. 5B depicts a technique for sharing eye tracking data between a client and a synthesizer. For clarity, the description of fig. 3 is described with reference to fig. 1 and 4 as described above. While the various processes shown in fig. 5B are shown in a particular order, it should be understood that the various processes described may be performed in a different order. Further, not all of the various processes need necessarily be performed. The flow diagram begins at block 555, where the client processes the frame and obtains eye tracking parameters. In some embodiments, the client renders the frame for presentation by the compositor. According to some embodiments, when rendering a frame, the client determines certain eye tracking parameters, such as gaze angle. As shown at block 560, in one optional step, the client processes the frame based on the historical synthesizer eye tracking parameters. That is, the compositor passes data from the most recently presented frame and relating to the current gaze angle and/or visibility region back to the client for processing of subsequent frames. The client then renders the frame at block 565. By utilizing the compositor to communicate to the visibility region of the client, the client may avoid rendering pixels outside the visibility region, thereby improving performance. By utilizing the gaze angle delivered by the compositor, the client may more appropriately render the frame for the user to gaze, thereby improving image quality.

Flow diagram 550 continues at 570, where the compositor presents the rendered frames and in the process, compositor eye tracking parameters are obtained. In particular, the synthesizer may perform the steps described above with reference to fig. 5A. For example, based on the visibility region, the compositor may obtain a gaze angle value and a pixel location or a replacement pixel location. In accordance with one or more embodiments, the gaze used to obtain the gaze angle is from a separate gaze tracker. This gaze angle, pixel location, and visibility region may be considered, alone or in some combination, as a synthesizer eye tracking parameter. According to some embodiments, the synthesizer generates frames from gaze angles and visibility regions using various eye tracking parameters.

According to some embodiments, the compositor may optionally process the current frame based on historical client data from the client rendered frame being presented by the compositor. As such, at block 575, the synthesizer optionally processes the current frame based on historical synthesizer eye tracking parameters. I.e. from one frame to another, eye tracking data, such as gaze angle and/or visibility region, may be utilized. In some implementations, doing so may reduce latency in processing consecutive frames.

Optionally, as shown at block 580, in some embodiments, the synthesizer processes the current frame based on historical client eye tracking parameters. That is, in some embodiments, the client may pass data used during rendering of the previous frame to the compositor for processing of additional frames. For example, the gaze angle used during rendering may be sent to the client along with the rendered frames. In this way, replacing pixel locations and gaze angles as described above with reference to fig. 4B and 5A will enable more efficient determination and/or improved latency, depending on the data being shared.

The flow chart continues at block 585. If there are additional frames to process, the flow chart returns to block 555 and the client optionally processes the additional frames based on the eye tracking parameters passed from the compositor until there are no additional frames to process.

FIG. 6 depicts a flow diagram for selecting an updated pixel location in accordance with one or more embodiments. In particular, flow diagram 600 illustrates an example technique for determining replacement pixel locations, according to some embodiments. For clarity, the description of fig. 6 is described with reference to fig. 1 and 4 as described above. While the various processes shown in fig. 6 are shown in a particular order, it should be understood that the various processes described may be performed in a different order. Further, not all of the various processes need necessarily be performed.

The flow diagram 600 begins at 605 where a pixel associated with a center of a field of view is identified within a visible region. The center of the field of view may be a pixel on the screen that is located in front of the eye and may be determined based on a vector that originates at the center of the eye and extends in a perpendicular manner to the surface of the screen. Thus, the center of the field of view may be positioned at or offset from: the center of the screen or the pixel associated with the center of the lens between the screen and the eye.

The flow diagram 600 continues at 610 where a vector is determined by the pixel associated with the center of the field of view and the pixel associated with the gaze direction. The pixels associated with the gaze direction may be determined as described above with reference to block 505 of flowchart 500. Flow diagram 600 ends at block 615, where a replacement pixel is selected from a set of pixels along the determined vector and within the visible region.

According to some embodiments, the system setup may be configured to track both eyes. If valid data is not available for one eye, the system may be configured to use the other eye to predict a fixation of the eye with invalid data. For example, gaze data for each eye may be obtained. If it is determined that one of the two sets of gaze data includes invalid data, a predictive model may be applied to the other set of gaze data to predict the missing gaze data. The set of gaze data may be determined to be invalid, e.g., based on hardware or software feedback, based on a check for data consistency between eyes, based on a threshold movement within an expected movement, etc. In some implementations, the gaze direction may be determined based on the content being presented. For example, if an object is presented to a user and it is determined that one eye is looking outside the object or within a threshold distance of the object, the gaze may be "clipped" to the object. Similarly, scene depth can be used to determine where the user is looking and to ensure that both eyes are looking at something at the same depth. In some implementations, the depth at which the eye is looking can be determined using an eye model and pupil diameter.

Turning to fig. 7, an electronic device 700 in accordance with one or more embodiments of the present disclosure is presented. The electronic device 700 may be part of a multi-function device, such as a mobile phone, a tablet, a personal digital assistant, a portable music/video player, a wearable device, a base station, a laptop computer, a desktop computer, a network device, or any other electronic device. Electronic device 700 is typically used for eye tracking, for example, in the context of XR applications. It should be understood that the various components of electronic device 700 may be distributed differently within the device or may be distributed over additional devices, such as additional client devices, network devices, etc.

The electronic device 700 may include a processor 710, such as a Central Processing Unit (CPU). Processor 710 may be a system on chip such as those found in mobile devices and may include one or more dedicated Graphics Processing Units (GPUs). Further, processor 710 may include multiple processors of the same or different types. The electronic device 700 may also include memory 720. Memory 720 may include one or more different types of memory that may be used to perform device functions in conjunction with processor 710. For example, the memory 720 may include a cache, a Read Only Memory (ROM), a Random Access Memory (RAM), or any kind of transitory or non-transitory computer-readable storage medium capable of storing computer-readable code. Memory 720 may store various programming modules that are executed by processor 710, including an eye tracking module 722 and other applications 724. The electronic device 700 may also include storage 730. Storage 730 may include one or more non-transitory computer-readable media including, for example, magnetic disks (fixed, floppy, and removable disks) and tape, optical media such as CD-ROMs and Digital Video Disks (DVDs), and semiconductor memory devices such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs). The storage 730 may include, for example, data for performing eye tracking techniques or data for presenting information to a user utilizing eye tracking techniques.

The electronic device 700 may also include one or more cameras 712 or other sensors, such as depth sensors from which scene depth may be determined. In one or more embodiments, each of the one or more cameras 712 may be a conventional RGB camera or a depth camera. Additionally, the cameras 712 may include stereo cameras or other multi-camera systems, time-of-flight camera systems, etc., that capture images from which depth information of a scene may be determined. In some implementations, the one or more cameras 712 can be cameras used for eye tracking. Electronic device 700 may allow a user to interact with the XR environment. There are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed for placement on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smart phones, tablets, and desktop/laptop computers. The head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment, and/or one or more microphones for capturing audio of the physical environment. The head mounted system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uuled, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display can be configured to be selectively opaque. Projection-based systems may employ retinal projection techniques that project a graphical image onto a person's retina. The projection system may also be configured to project the virtual object into the physical environment, for example as a hologram or on a physical surface.

Referring now to fig. 8, a simplified functional block diagram of an illustrative multi-function electronic device 800 is shown, according to one embodiment. Each of the electronic devices may be a multi-function electronic device, or may have some or all of the components described for a multi-function electronic device described herein. Multifunction electronic device 800 can include processor 805, display 810, user interface 815, graphics hardware 820, device sensors 825 (e.g., proximity/ambient light sensors, accelerometers, and/or gyroscopes), microphone 830, audio codec 835, speaker 840, communications circuit 845, digital image capture circuit 850 (e.g., including a camera system), video codec 855 (e.g., supporting a digital image capture unit), memory 860, storage 865, and communications bus 870. The multifunction electronic device 800 may be, for example, a digital camera or a personal electronic device such as a Personal Digital Assistant (PDA), a personal music player, a mobile phone, or a tablet computer.

The processor 805 may execute instructions necessary to carry out or control the operation of many of the functions performed by the device 800 (e.g., the generation and/or processing of images such as those disclosed herein). The processor 805 may, for example, drive the display 810 and may receive user input from the user interface 815. The user interface 815 may allow a user to interact with the device 800. For example, user interface 815 may take a variety of forms, such as a button, a keypad, a dial, a click wheel, a keyboard, a display screen, and/or a touch screen. The processor 805 may also be, for example, a system-on-chip, such as that found in mobile devices, and include a dedicated GPU. The processor 805 may be based on a Reduced Instruction Set Computer (RISC) or Complex Instruction Set Computer (CISC) architecture, or any other suitable architecture, and may include one or more processing cores. The graphics hardware 820 may be dedicated computing hardware for processing graphics and/or assisting the processor 805 in processing graphics information. In one implementation, the graphics hardware 820 may include a programmable GPU.

The image capture circuitry 850 can include two (or more) lens assemblies 880A and 880B, where each lens assembly can have a separate focal length. For example, lens assembly 880A may have a shorter focal length than lens assembly 880B. Each lens assembly may have a separate associated

sensor element

890A or 890B. Alternatively, two or more lens assemblies may share a common sensor element. The image capture circuit 850 may capture still images and/or video images. The output from image capture circuitry 850 may be at least partially processed by: a video codec 855 and/or a processor 805 and/or graphics hardware 820, and/or a dedicated image processing unit or pipeline incorporated within the circuitry 850. The captured images may be stored in memory 860 and/or storage 865.

The sensor and camera circuitry 850 may capture still images and video images that may be processed at least in part by the following devices in accordance with the present disclosure: a video codec 855 and/or processor 805 and/or graphics hardware 820, and/or a dedicated image processing unit incorporated within the circuit 850. Such captured images may be stored in memory 860 and/or storage 865. The memory 860 may include one or more different types of media used by the processor 805 and the graphics hardware 820 to perform device functions. For example, memory 860 may include memory cache, ROM, and/or RAM. Storage 865 may store media (e.g., audio files, image files, and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage 865 may include one or more non-transitory computer-readable storage media including, for example, magnetic disks (fixed, floppy, and removable disks) and tape, optical media such as CD-ROMs and Digital Video Disks (DVDs), and semiconductor memory devices such as electrically programmable read-only memories (EPROMs) and electrically erasable programmable read-only memories (EEPROMs). Memory 860 and storage 865 may be used to tangibly retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. Such computer program code may, when executed by, for example, the processor 805, implement one or more of the methods described herein.

A physical environment refers to the physical world that people can sense and/or interact with without the aid of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, an augmented reality (XR) environment refers to a fully or partially simulated environment in which people sense and/or interact via electronic devices. For example, the XR environment may include Augmented Reality (AR) content, mixed Reality (MR) content, virtual Reality (VR) content, and so on. In the case of an XR system, a subset of the person's physical movements, or a representation thereof, is tracked and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one laws of physics. For example, the XR system may detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to the manner in which such views and sounds change in the physical environment. As another example, the XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting the XR environment and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds would change in the physical environment. In some cases (e.g., for accessibility reasons), the XR system may adjust characteristics of graphical content in the XR environment in response to representations of physical motion (e.g., voice commands).

It is to be understood that the above description is intended to be illustrative, and not restrictive. The materials have been presented to enable any person skilled in the art to make and use the claimed disclosed subject matter and to provide such materials in the context of particular embodiments, variations of which will be apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with one another). Accordingly, the particular arrangement of steps or acts illustrated in fig. 2-3 and 5-6 or the arrangement of elements illustrated in fig. 1, 4 and 7-8 should not be construed as limiting the scope of the disclosed subject matter. The scope of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms "including" and "in which" are used as the plain-english equivalents of the respective terms "comprising" and "wherein.

Claims

1. A method, the method comprising:

determining an initial pupil position of a user relative to a lens located in front of the user;

detecting a change in pupil position relative to the lens to an updated pupil position relative to the lens;

determining that the updated pupil location is outside of a bounding box associated with the lens;

replacing the updated pupil position with a replacement pupil position within the bounding box associated with the lens; and

using the updated pupil position for an eye tracking function.

2. The method of claim 1, wherein the initial pupil position is associated with a first frame presented to the user, the method further comprising:

presenting a second frame to the user based on the updated pupil position.

3. The method of claim 2, wherein the second frame is distorted according to the updated pupil position.

4. The method of claim 2, wherein replacing the updated pupil location with a replacement pupil location is determined by applying a twitch function from the initial pupil location to the replacement pupil location.

5. The method of claim 4, wherein the bradyseis function comprises a time-based bradyseis function, and wherein the replacement pupil location is reached during a series of frames including the second frame.

6. The method of claim 1, wherein the bounding box is determined from calibration information for the lens.

7. The method of claim 1, wherein the bounding box is determined according to a predetermined area of the lens.

8. A non-transitory computer-readable medium comprising computer-readable code executable by one or more processors to:

replacing the updated pupil location with a replacement pupil location that is within the bounding box; and

using the updated pupil position for an eye tracking function.

9. The non-transitory computer-readable medium of claim 8, wherein the initial pupil position is associated with a first frame presented to the user, and further comprising computer-readable code to:

presenting a second frame to the user based on the updated pupil position.

10. The non-transitory computer-readable medium of claim 9, wherein the second frame is distorted according to the updated pupil location.

11. The non-transitory computer-readable medium of claim 9, wherein the computer-readable code for replacing the updated pupil location with a replacement pupil location is determined by applying a bradyseis function from the initial pupil location to the replacement pupil location.

12. The non-transitory computer-readable medium of claim 11, wherein the bradykinetic function comprises a time-based bradykinetic function, and wherein the replacement pupil location is reached during a series of frames including the second frame.

13. The non-transitory computer readable medium of claim 8, wherein the bounding box is determined from calibration information for the lens.

14. The non-transitory computer readable medium of claim 8, wherein the bounding box is determined according to a predetermined area of the lens.

15. A system, the system comprising:

one or more processors; and

one or more computer-readable media comprising computer-readable code executable by the one or more processors to:

using the updated pupil position for an eye tracking function.

16. The system of claim 15, wherein the initial pupil position is associated with a first frame presented to the user, and further comprising computer readable code to:

presenting a second frame to the user based on the updated pupil position.

17. The system of claim 16, wherein the second frame is distorted according to the updated pupil position.

18. The system of claim 16, wherein the computer readable code for replacing the updated pupil location with a replacement pupil location is determined by applying a twitch function from the initial pupil location to the replacement pupil location.

19. The system of claim 18, wherein the bradyseis function comprises a time-based bradyseis function, and wherein the replacement pupil location is reached during a series of frames comprising the second frame.

20. The system of claim 15, wherein the bounding box is determined from calibration information for the lens.