KR101950641B1 - Scene analysis for improved eye tracking - Google Patents

Abstract

Techniques for scene analysis for improved eye tracking are generally described. In some instances, the detected gaze targets may be derived from gaze direction information from the ocular-facing sensor. The detected gaze target positions and / or motion may be enhanced by capturing and analyzing digital scene information from a scene viewed by the eye. The digital scene information captured by the digital camera may be analyzed to identify potential gaze targets, such as gazed gazed targets, moving gaze targets, and / or gaze targets that accelerate. The detected gaze targets may be changed to the positions of the selected gaze targets.

Description

[0001] SCENE ANALYSIS FOR IMPROVED EYE TRACKING [0002]

The present disclosure generally describes techniques, including devices, methods, and computer readable media for scene analysis for improved eye tracking.

Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not considered to be prior art by inclusion in this section.

Eye tracking systems may include miniature cameras capable of tracking human eye motions. Eye tracking has a variety of useful and interesting applications. For example, eye tracking can be used efficiently in a variety of medical and industrial applications, as well as augmented reality (AR) applications. Eye tracking also has a variety of potential User Interface (UI) applications such as, for example, allowing the device user's gaze to control the selection pointer. A device user may gaze at a virtual object such as a file, press a button, then look at the destination location for that object, and release the button to move the object. Eye tracking also presents opportunities to gather user interest and user activity information, for example, whether users have seen the advertisement. These and other uses for eye tracking offer many interesting possibilities.

However, ocular tracking techniques remain relatively early in development and there are many challenges to deal with as ocular tracking develops.

The present disclosure generally describes techniques, including devices, methods, and computer readable media for scene analysis for improved eye tracking. Some exemplary eye-tracking methods performed by a computing device include receiving eye-gaze direction information from an eye-facing sensor; Receiving digital scene information from a digital camera, the digital camera being capable of being oriented to capture digital scene information from a scene viewed by an eye; receiving the digital scene information; Determining a detected visual target in the digital scene information based on the visual direction information; Within the digital scene information, analyzing the digital scene information to identify potential gaze targets, the potential gaze targets may include, for example, stationary gaze targets, moving gaze targets, and / Analyzing the digital scene information; Selecting a potential gaze target; And / or changing the detected line-of-sight target to the position of the selected line-of-sight target.

Computing devices and computer readable media having instructions for implementing the various techniques described herein are also disclosed. Exemplary computer-readable media may comprise non-transitory computer-readable storage media having computer-executable instructions executable by a processor, the instructions, when executed by the processor, cause the processor to perform the various methods To perform any combination of the above. Exemplary computing devices include a computing device having a computing device with an eye tracking accuracy enhancer configured to perform, for example, a digital camera, an eye-facing sensor, and any combination of the various methods provided herein. Mounting devices.

The foregoing summary is exemplary and is not intended to be limiting in any way. In addition to the exemplary aspects, embodiments, and features described above, additional aspects, embodiments, and features will become apparent with reference to the drawings and the following detailed description.

The foregoing and other features of the disclosure will become more fully apparent from the following description and the appended claims taken in conjunction with the accompanying drawings. With the understanding that these drawings depict only some embodiments in accordance with the present disclosure and are therefore not to be considered limiting of its scope, the present disclosure will be described in further detail and detail through the use of the accompanying drawings, in which , All arranged in accordance with at least some embodiments of the disclosure:
1 is a diagram illustrating an exemplary head-mounted device worn by a user;
2 is a diagram illustrating a scene including potential gaze targets, a detected gaze target, and its motion;
3 is a block diagram of a computing device as an example of a computer integrated within a head mounting device;
4 is a flow diagram illustrating an exemplary eye tracking method;
5 is a block diagram illustrating an exemplary eye tracking accuracy enhancer;
Figure 6 is a diagram illustrating a scene including potential gaze targets, detected gaze targets, selected gaze targets, and AR information for a selected gaze target;
FIG. 7 is a diagram illustrating a scene including potential gaze targets, a detected gaze target, and a selected gaze target, wherein the selected gaze target may be applied as a user input for controlling at least one computing device function.

In the following detailed description, reference is made to the accompanying drawings which form a part thereof. In the drawings, similar symbols generally identify similar components, unless the context otherwise requires. The illustrative embodiments set forth in the description, the drawings, and the claims are not meant to be limiting. Other embodiments may be utilized and other changes may be made without departing from the spirit or scope of the disclosure herein. Aspects of the disclosure may be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are expressly contemplated and are part of the disclosure, as generally described herein and illustrated in the drawings It will be easy to understand.

This disclosure is generally directed to techniques involving methods, devices, systems and / or computer-readable media that are used effectively in the present description of scene analysis for improved eye tracking, among others . In some instances, the gaze targets may be derived from gaze direction information from an eye-facing sensor. The gaze target positions and / or motion may be improved by capturing and analyzing digital scene information from a scene viewed by the eye. Digital scene information captured by the digital camera may be analyzed, for example, to identify potential gaze targets, such as stationary gaze targets, moving gaze targets, and / or accelerating gaze targets. The detected gaze targets may be modified to include the positions of the selected potential gaze targets, also referred to herein as " selected gaze targets ".

In some embodiments, the techniques described herein may be incorporated into head mounted devices, such as, for example, GOOGLE GLASS® type devices. Such devices may, for example, provide a transparent lens through which the user may view the physical world. The devices may further display digital objects and information on the lens. Devices may allow users, such as wearers of head mounted devices, to interact with their objects and information as well as select digital objects and information to be displayed. Devices may optionally be connected to computer networks by, for example, wirelessly connecting to the Internet. The devices may also include integrated digital cameras, which may allow the users to capture digital pictures and videos of the scenes as they have viewed through the lens. In addition to these and other features as may be included in the exemplary devices, the devices may incorporate eye-to-face sensors and may be prepared to perform eye tracking methods such as those described herein .

Eye-to-face sensors may include expensive sensors with high accuracy, or low-cost, low-accuracy sensors. Embodiments of the disclosure may involve any type of eye-facing sensor. Highly accurate sensors may include, for example, cameras that capture high resolution visible light images of the eye, and eye tracking systems may perform relatively more processing to achieve greater accuracy of the calculated eye position have. Low-cost, low-accuracy sensors may include, for example, cameras that capture low-resolution infrared (IR) line images of the eye, and eye tracking systems may be used to achieve comparatively smaller accuracy of the calculated eye position Relatively less extensive processing may be performed.

In eye tracking, there are fundamental difficulties that cause difficulties in solving smooth balls, very small and low contrast rotations in the eye. One problem with some eye-to-face sensors and corresponding eye tracking systems is that the reflected IR line of moderate intensity and contrast may cause a jumpy estimation of inaccurate eye directions, Which leads to difficult motions. While high resolution visible light eye tracking systems may be much better, these eye tracking systems utilize more expensive eye-to-face sensor cameras and more processing power, and do not function in low light conditions.

The difficulties of eye tracking are particularly expressed in the context of physical environments rather than virtual environments. In physical environments, current techniques do not provide any information about eye-interesting points or other potential eye-gaze targets that are of interest in the physical scenes as viewed. Thus, in some embodiments, the techniques described herein may be used to capture and analyze digital scene information from physical views seen by an eye, identify potential gaze targets, and use these identified potential gaze targets Eye target positions and / or eye target motion derived from eye-to-face sensors.

In some embodiments, the eye tracking systems compare the detected eye tracking position, velocity, and / or acceleration values with estimated eye tracking values corresponding to the potential eye targets of the physical world, Lt; / RTI > may also produce an improved eye-tracking target estimate, also referred to as an eye-tracking target estimate. Potential line of sight targets may be identified, for example, in real-time video in a physical world scene.

The potential gaze targets may be identified, for example, by using object recognition, or by applying any of a variety of graphics analysis algorithms to identify potential gaze targets, or by identifying object motion in, for example, a scene have. Embodiments may subsequently determine position, velocity, and / or acceleration values for each identified potential eye target, and may compare these determined values to detected eye tracking position, velocity, and / or acceleration values .

Embodiments may also select an appropriate potential gaze target that includes position, velocity, and / or acceleration values that substantially match the detected ocular tracking position, velocity, and / or acceleration values. Embodiments may then infer that the precise eye tracking position, velocity, and / or acceleration values are those of the selected gaze target. The modified, improved eye tracking position, velocity, and / or acceleration values may be stored or otherwise used for any application that uses eye tracking information, for example.

Embodiments may suitably perform data transformations to enable comparisons between detected eye tracking position, velocity, and / or acceleration values and potential eye target position, velocity, and / or acceleration values. For example, in some embodiments, the detected eye tracking position values may include angular coordinates corresponding to the ocular gaze direction. On the other hand, the potential gaze target position values may include, at least initially, e.g., two-dimensional x, y coordinates in the plane of the captured video frame. The detected eye tracking angular coordinates may be converted to two dimensional x, y coordinates, for example, to support comparison operations using geometric relationships between camera position and eye position, or vice versa. It will also be appreciated that embodiments may convert all information to any intermediate coordinate system when desired to support comparison operations.

In some embodiments, the eye tracking systems may compare the detected line-of-sight targets with the positions of the stationary potential line-of-sight targets in the physical scene and may appropriately compare the detected line-of-sight targets to the positions of the stationary potential line- To correct the detected eye targets within their noise boundaries. In some embodiments, the eye tracking systems may compare the detected line-of-sight directions to the estimated line-of-sight directions corresponding to the positions of the stationary potential line-of-sight targets in the physical scene, and compare the detected line-of- To correct the detected line-of-sight directions within their noise boundaries.

In some embodiments, the eye tracking systems may include detected gaze target motion, which may include gaze target velocity and / or gaze target acceleration, and may be derived, for example, from a series of detected gaze targets within a physical scene ) May be compared to the line of sight target motion modeled to follow the moving potential eye targets in the physical scene, and the embodiments may compare the detected line of sight target motion to the modeled eye gaze motion following the moving potential eye targets in the physical scene The line-of-sight target motion detected by changing to the line-of-sight target motion may be corrected within the noise boundary. In some embodiments, the eye tracking systems can be used to detect detected eye movements (which may be derived, for example, from a series of detected eye orientations) Eye movements and embodiments may change the detected eye motion to eye movement that is modeled to follow moving potential eye targets in the physical scene or to potential eye targets that match the modeled eye motion Thereby correcting the detected eye motion within the noise boundary.

Exemplary embodiments may capture images of the physical world as viewed by the user, or an approximation thereof. For example, embodiments may capture video using a digital camera proximate to the user's eye. Embodiments may analyze captured physical world images to extract potential eye targets of the physical world. This analysis may include object recognition in some embodiments. In some embodiments, any of the various algorithms may be adapted to identify, for example, sets of related graphical features that may be related, for example, by color, proximity, texture, and the like. Exemplary algorithms include Speeded Up Robust Features (SURF), Scale Invariant Feature Transform (SIFT), and Kanade-Lucas-Tomasi (KLT) type algorithms. Each identified set of related graphical features may be considered as a potential gaze target and by tracking the positions of each identified set of related graphical features across multiple video frames, And may establish the velocities and accelerations of the targets.

Embodiments may calculate velocity information for moving potential target targets over a plurality of video images and for each identified potential gaze target by computing a first derivative of the position information . Embodiments can also calculate acceleration information for potential gaze targets that accelerate by computing a second derivative of the position information across multiple video images and for each identified potential gaze target have. Embodiments may also calculate linear and temporal derivative information for the detected line of sight targets and / or line of sight directions for comparisons with potential gaze target velocities and accelerations. Embodiments may calculate linear and logarithmic derivative information using any of a variety of mathematical approaches, as will be appreciated, to generate comprehensive motion flows suitable for comparison operations.

Embodiments may further include detecting eye tracking data by inferring intended eye directions or eye targets in a physical world having positions, velocities, and / or accelerations that substantially match the detected eye tracking data attributes And may be changed to those of potential gaze targets. For example, embodiments may include a substantially matching predicted gaze target position, an expected gaze target velocity, and / or an expected gaze target position that substantially match the detected gaze target positions and / And may select an identified potential gaze target having an acceleration. Embodiments may change the detected gaze targets to generate modified, accuracy-enhanced gaze target information that includes the positions of the selected gaze targets.

Figure 1 is a diagram illustrating an exemplary head-mounted device that a user wears, arranged in accordance with at least some embodiments of the present disclosure. Figure 1 illustrates a user 150, a head-mounted device 100 worn by a user 150, a detected gaze target 141, and a selected gaze target 142. The head-mounted device 100 includes a computer 101, a digital camera 102, and an eye-facing sensor 103. The head mounting device 100 also includes a lens or other view optics 110, but the view optics 110 may be omitted (not included) in some embodiments. In FIG. 1, the eye-to-face sensor 103 may be adapted to detect real-time eye-gaze direction information for the eye of the user 150. The digital camera 102 may be configured to capture real-time digital scene information from a scene viewed by at least one eye of the user 150, e.g., a scene including a detected gaze target 141 and a selected gaze target 142 Or may be oriented. An exemplary scene is illustrated in FIG.

2 is a diagram illustrating a scene including potential gaze targets, a detected gaze target, and its motion, arranged in accordance with at least some embodiments of the present disclosure. 2 illustrates a scene 200 that may be captured, for example, as real-time video by a digital camera 102. [ The scene 200 includes a potential gaze target 201, a potential gaze target 202, a potential gaze target 203, and a potential gaze target 204.

The potential gaze target 201 may have the motion M (F1) illustrated by the motion vector extending from the potential gaze target 201. [ The potential gaze target 201 is an exemplary moving gaze target within the scene 200, and the potential gaze target 201 may, for example, show a relatively constant velocity and no acceleration. The potential gaze target 202 may have a motion M (F2). The potential gaze target 202 may be an exemplary static gaze target within the scene 200 and the potential gaze target 202 may show substantially no velocity or no acceleration. The potential gaze target 203 may have a motion M (F3), and the potential gaze target 204 may have a motion M (F4). Potential gaze targets 203 and 204 are exemplary accelerating gaze targets within scene 200 and potential gaze targets 203 and 204 may show changes in velocities.

The four potential gaze targets 201-204 are illustrated within scene 200, at exemplary positions, and with exemplary motions. However, scenes may, as will be appreciated, include any number of potential gaze targets in any positions and in any motion. The term " motion " refers to velocity, acceleration, or both, as used herein.

FIG. 2 also shows a graphical representation of the detected gaze position at the exemplary eye target position within the scene 200, illustrated by the motion vectors extending from the detected gaze target 141, The target 141 is illustrated. M (DT) may, for example, show an acceleration in the scene 200 that includes a change in velocity in accordance with the illustrated motion vectors. However, the line of sight target 141 is not part of the scene 200 as captured by the digital camera 102, and the line of sight target 141 position and motion in the scene 200 is in the eye-to-face sensor 103 And may be derived from the detected real-time gaze direction information. 2 also illustrates a selected line of sight target 142 within the scene 200, where the computer 101 may select, for example, a potential line of sight target 204 as the selected line of sight target 142.

The computer 101 may also have an eye tracking accuracy enhancer stored in memory and executable by the processor, for example, as described in connection with FIG. In some embodiments, the eye tracking accuracy enhancer comprises: receiving real-time-of-sight direction information from eye-facing sensor 103; Receive real-time digital scene information (e.g., from scene 200) from digital camera 102; Speed and / or acceleration from the scene 200, for example, by determining the position, velocity, and / or acceleration of the detected line of sight target 141 within the scene 200 based on the line of sight direction information from the eye- Determine a detected gaze target 141 in the scene information; Analyzing real-time digital scene information from scene 200 to identify potential gaze targets 201-204; Selects a potential gaze target 204 from the identified potential gaze targets 201-204; And to change the line of sight target 141 to include at least the position (s) of the selected line of sight target 142. Once the selected gaze target 142 is established, the selected gaze target 142 may determine the AR information for various applications, e.g., display, and / or determine the selected gaze target 142 to the computing device 101, Or as a user input to control the functions of the device.

In some embodiments, determining the detected gaze target 141 in the real-time digital scene information from the scene 200 based on the gaze direction information from the eye- ) To apply mathematical principles such as trigonometry, geometry, etc. For example, the computer 101 may calculate the x, y coordinates of the detected gaze target 141 in a coordinate system having a center point (x = 0, y = 0) in the center of the scene 200. Embodiments may use, for example, tangent or cotangent functions, or mathematical equivalents thereof, wherein the x and y components of the detected viewing direction information [theta] _x and [theta] _y are known triangle leg distances (D), for example, the distance from the eye of the user 150 to the view optics 110, respectively. In some embodiments, determining the detected gaze target 141 in the real-time digital scene information from the scene 200 based on the gaze direction information from the eye-facing sensor 103 may include determining the gaze direction, E. G., Eye angles, and corresponding gaze target positions, where embodiments may look up gaze target positions corresponding to the detected gaze directions. Moreover, embodiments may also determine gaze target velocities and / or accelerations based on gaze target vector histories derived, for example, from gaze target positions across a plurality of video frames captured from scene 200 .

In some embodiments, the gaze target velocities and / or accelerations may be compared using an eye mechanical model simulator. The motion of the eye is a secondary spring-mass system driven by a feedback loop with approximately delay. This may be represented as visible rise times that lag and overshoot the object that accelerates when the targets change speed. Thus, for example, M (DT) in scene 200 may be determined by the user 150 (e.g., a user), rather than using a " raw " M (DT) function that strictly applies the detected viewing direction for comparison And may be adjusted by operation of the ocular machine model simulator to reflect possible motions of the potential gaze target to be imaged.

As described herein, some embodiments may operate in the ocular coordinate space rather than the captured scene coordinate space. For example, instead of determining the detected gaze target 141 within the scene 200, the embodiments may determine the expected gaze directions associated with the potential gaze targets 201-204. These embodiments may also be used to simulate eyeball movements expected to follow each of the potential eye targets 201-204, including, for example, lag, overshoot, and / or other mechanical artifacts. A model simulator may also be used. Regardless of the technical approach, the embodiments compare the ocular tracking information from ocular-facing sensor 103 and its variations to the positions, velocities and / or accelerations of potential gaze targets 201-204, A potential gaze target within the scene 200 that may be the most likely to be watched by the viewer 150 may be selected.

The computer 101 may be configured to determine the potential gaze targets 201-204 in the image or video data within the scene 200, such as SURF, SIFT, and / or KLT type algorithms as disclosed herein, Various algorithms may be applied. In some embodiments, the computer 101 may apply object recognition to analyze the digital scene information from the scene 200 and thereby identify objects that may be potential gaze targets 201-204. In some embodiments, the computer 101 may analyze the digital scene information from the scene 200 by at least partially identifying sets of related graphical features, and each set of related graphical features may include, for example, Target, such as one of the potential gaze targets 201-204.

The computer 101 may compare the detected gaze target 141 position, velocity, and / or acceleration with the positions, velocities, and / or accelerations of the potential gaze targets 201-204, for example The gaze target 142 selected from the potential gaze targets 201 to 204 may be selected. For example, in some embodiments, the selected gaze target 142 may have a feature position within the digital scene information from the scene 200, which may be within the error boundaries surrounding the detected gaze target 141 It is possible. The error bounds may depend on the accuracy of the eye-to-face sensor 103, the alignment between the camera image and the eye, and / or the calculations used to map the detected eye target 141 to the scene 200. The error bounds may define a region surrounding the detected gaze target 141 where the detected gaze target 141 may change. In some embodiments, the selected gaze target 142 may further include a moving gaze target having a gaze target velocity that substantially matches the detected gaze target 141 velocity. In some embodiments, the selected gaze target 142 may further include an accelerating gaze target having a gaze target acceleration substantially matching the detected gaze target 141 acceleration.

In some potential real world scenarios, the various potential gaze targets may show speeds and / or accelerations that are similar to at least the detected gaze target velocities and / or accelerations, as well as positions within the visual target error boundaries. Accordingly, embodiments may employ techniques for selecting an appropriate potential target of sight among the various possible candidates. The computer 101 may dynamically assign weights to a plurality of potential gaze targets 201-204, for example. The computer 101 may use dynamically assigned weights to select a potential gaze target. For example, the computer 101 may assign weights to each of the potential gaze targets 201-204 according to their proximity to the detected gaze target 141, where the stronger weights are closer To the potential gaze targets that are close to the target. The computer 101 may assign weights to each of the potential gaze targets 201-204 according to the degree of match between the potential gaze target velocities and / or accelerations and the velocity and / or acceleration of the detected gaze target 141 Where the stronger weights are applied to potential gaze targets having more closely matching velocities and / or accelerations. The strongest weighted potential gaze target may then be selected as the selected gaze target 142.

In some embodiments, the acceleration matches may be associated with relatively stronger ranges of weights, while the velocity matches may be associated with intermediate range weights, and the position matches may be associated with a relatively weaker range of weights May be associated. The computer 101 may thereby increase the accuracy of the potential gaze target selection when the potential gaze targets indicate motions that match the detected motion of the gaze target 141. [

In some embodiments, the computer 101 may be configured to display real-time gaze direction information and real-time digital scene information in real-time to continuously output selected gaze targets, such as the gaze target 142 selected in real- Time gaze direction information from the eyeball-facing sensor 103 and real-time digital scene information from the digital camera 102. [ In other embodiments, the computer 101 may be configured to determine selected gaze targets that may still prove useful to applications other than real-time, i.e., not later than in real time, It will be appreciated that information may be stored and processed.

In some embodiments, the digital camera 102 may be adapted to capture video at a plurality of different frame rates or resolutions. For example, the digital camera 102 may have a default frame rate and / or a default resolution, but the computer 101 may use a digital camera 102 to operate the digital camera 102 at different frame rates and / 102) default settings. The computer 101 may be adapted to adjust the digital camera 102 frame rates and / or resolutions appropriately for use in connection with capturing digital scene information in accordance with the present disclosure. For example, the computer 101 may operate the head-mounted device 100 by operating the digital camera 102 at a reduced frame rate and / or a reduced resolution, e.g., a frame rate or resolution lower than the default frame rate or resolution, Lt; RTI ID = 0.0 > battery life < / RTI > In applications where power is not limited and / or where more than any other accuracy is critical, the computer 101 may use a higher frame rate and / or an improved resolution, e.g., a higher frame rate or resolution than the default frame rate or resolution May be adapted to operate the digital camera 102. In some embodiments, the computer 101 may be configured to change the frame rate and / or resolution according to the speed and frequency of movement of the user ' s eyeball and / or the speed and frequency of movement of potential eye targets within the environment or environment It may be adapted.

3 is a block diagram of a computing device as an example of a computer integrated within a head mounted device, arranged in accordance with at least some embodiments of the present disclosure. In a very basic configuration 301, computing device 300 may include one or more processors 310 and system memory 320. The memory bus 330 may be used to communicate between the processor 310 and the system memory 320.

Depending on the desired configuration, the processor 310 may be of any type including, but not limited to, a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP) have. The processor 310 may include one or more levels of caching, such as a level 1 cache 311 and a level 2 cache 312, a processor core 313, and registers 314. The processor core 313 may comprise an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. The memory controller 315 may also be used with the processor 310 or, in some implementations, the memory controller 315 may be an internal portion of the processor 310. [

Depending on the desired configuration, the system memory 320 may be any type of memory including, but not limited to, volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) . The system memory 320 typically includes an operating system 321, one or more applications 322, and program data 325. In some embodiments, the operating system 321 may include a virtual machine managed by a virtual machine manager (VMM). Applications 322 may include, for example, eye tracking accuracy enhancer module (s) 323 as described herein. The program data 325 may include, for example, real-time gaze direction information 326 as may be received from the eye-to-face sensor 103, for example, real-time digital Scene information 327, and potential gaze targets 328 that may be extracted from the real-time digital scene information 327, respectively.

The computing device 300 may have additional features or functionality and additional interfaces to facilitate communications between the basic configuration 301 and any desired devices and interfaces. For example, the bus / interface controller 340 may be utilized to facilitate communications between the base configuration 301 and the one or more data storage devices 350 via the storage interface bus 341. The data storage devices 350 may be removable storage devices 351, non-removable storage devices 352, or a combination thereof. Examples of removable storage and non-removable storage devices include, but are not limited to, magnetic disk devices such as flexible disk drives and hard-disk drives (HDD), compact disk (CD) drives, or digital versatile disk , Solid state drives (SSD), and tape drives. Exemplary computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data You may.

Level 1 cache 311, level 2 cache 312, system memory 320, removable storage device 351, and non-removable storage device 352 are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices , Or any other medium that may be used to store the desired information and which may be accessed by computing device 300. Any such computer storage media may be part of the device 300.

The computing device 300 also facilitates communication from the various interface devices (e.g., output interfaces, peripheral interfaces, and communication interfaces) through the bus / interface controller 340 to the base configuration 301 (Not shown). Exemplary output devices 360 include a graphics processing unit 361 and an audio processing unit 362 that are connected to various external devices such as a display or speakers via one or more A / V ports 363 Lt; / RTI > Exemplary peripheral device interfaces 370 may include a serial interface controller 371 or a parallel interface controller 372 that may be connected to the digital camera 102 through an I / O port 373, (E.g., a printer, a scanner, etc.) and / or other output devices (e.g., keyboard, mouse, pen, voice input device, touch input device, Or may be configured to communicate over wired or wireless connections with the same devices. Other conventional I / O devices, such as a mouse, keyboard, etc., may of course be connected. Exemplary communication device 380 includes a network controller 381 that is arranged to facilitate communications with one or more other computing devices 390 through network communication via one or more communication ports 382 It is possible.

Computer storage media may be an example of communication media. Communication media may typically be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, including any information delivery media You may. A " modulated data signal " may be a signal that has been set or changed in such a manner as to cause one or more of its characteristics to encode information in the signal. By way of example, and not limitation, communication media may include wired media, such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared (IR) .

The computing device 300 may be embodied as a computer integrated within a head mounted device as illustrated in FIG. The computing device 300 is external to the head-mounted device as illustrated in FIG. 1, but yet is in wired or wireless communication with such a head-mounted device to receive real-time gaze direction information 326 and real-time digital scene information therefrom Lt; / RTI > computer. For example, the computing device 300 may be implemented as a personal or business computer, including both laptop computers and non-laptop computer configurations, or servers within a data center. Embodiments in which the computing device 300 operates in a manner other than real-time, using visual direction information and digital scene information that do not exist in real-time, may also be designed in accordance with the present disclosure.

Figure 4 is a flow diagram illustrating an exemplary eye tracking method arranged in accordance with at least some embodiments of the present disclosure. Exemplary flow diagrams illustrate operations such as may be performed in a method, functional modules within computing device 300, and / or blocks that represent instructions, such as may be written on computer readable medium 450 / RTI > may include one or more operations / modules as illustrated by blocks 401-407.

In FIG. 4, blocks 401 to 407 are illustrated as including blocks that are sequentially executed, for example, with block 401 as the first and block 407 as the last. It will be appreciated, however, that these blocks may conveniently be rearranged to suit particular embodiments and that these blocks or portions thereof may be performed concurrently in some embodiments. It will also be appreciated that, in some instances, the various blocks may be removed, partitioned into additional blocks, and / or combined with other blocks.

FIG. 4 illustrates an exemplary method by which computing device 300 may perform eye tracking, wherein eye tracking data from an eye-to-face sensor may be used to perform scene analysis, to select potential eye targets, May be improved to increase its accuracy by changing the tracking data.

At a block 401, the computing device 300 receives information from at least one eye-to-face sensor 103, such as at least one eye, for example, eye direction information about the eye of the user 150 . Computing devices including integrated eye-to-face sensors may receive eye-gaze direction information from integrated eye-to-face sensors. Computing devices communicatively coupled with external eye-facing sensors may optionally receive eye-direction information from external eye-facing sensors via any number of intermediate devices or network infrastructures. Block 401 may be followed by block 402.

At block 402, the computing device 300 may receive digital scene information from the digital camera 102. The digital camera 102 may be oriented to capture digital scene information from a scene viewed by the eye of the user 150 that may be tracked at block 401. [ Computing devices, including integrated digital cameras, may receive digital scene information from integrated digital cameras. Computing devices communicatively coupled with external digital cameras may optionally receive digital scene information from external digital cameras via any number of intermediate devices or network infrastructures. In some embodiments, the computing device 300 may adjust the frame rate and / or resolution of the digital camera 102 at block 402, e.g., the digital camera 102 may have a default frame rate or a default resolution , And the computing device 300 may operate the digital camera 102 at different frame rates and / or different resolutions. Block 402 may be followed by block 403.

In block 403, the computing device 300 determines, based on the gaze direction information received in block 401, the gaze position in the digital scene information, the velocity, and / The target may be determined. Determining the line-of-sight target may include, for example, determining line-of-sight targets in a number of different video frames and determining a line-of-sight target position, And / or < / RTI > In some embodiments, determining the line-of-sight target may include applying an eye-machine model simulator to simulate motion of the object in the digital scene information received at block 402 based on the detected eye movements. Some embodiments may omit gaze target determinations and may instead use, for example, eye angular orientations, eye angular velocities < RTI ID = 0.0 > And eye angular accelerations. Block 403 may be followed by block 404.

At a block 404, the computing device 300 may analyze the potential gaze targets and their potential gaze targets in the digital scene information received at block 402. " Analyze the digital scene to identify potential gaze targets and their positions, velocities, and / Lt; RTI ID = 0.0 > digital scene information. ≪ / RTI > For example, the computing device 300 may apply object recognition and / or graphics analysis algorithms to the digital scene information received at block 402, as disclosed herein. The identified potential gaze targets may include, for example, stationary gaze targets, moving and / or accelerating gaze targets. The computing device 300 may track the positions of potential gaze targets across a number of different video frames and may determine the corresponding gaze targets and / or accelerations to determine potential gaze target positions, velocities, and / It is also possible to calculate the derivative derivatives. In some embodiments, the potential gaze target positions, velocities, and / or accelerations are changed to, for example, each coordinate system or eye coordinate space used for the viewing direction in the gaze direction information received at block 401 It is possible. An ocular machine model simulator may be used to simulate eye movements based on potential gaze target positions, velocities, and / or accelerations. Block 404 may be followed by block 405.

At block 405, the computing device 300 may select a feature from among the features identified at block 404. ≪ RTI ID = 0.0 > [0040] < / RTI & In some embodiments, the computing device 300 may select a potential gaze target that may have a position, as determined at block 404, which may include an error boundary surrounding the detected gaze target, as determined at block 403 . In some embodiments, the plurality of detected gaze target positions may optionally be compared to a number of potential gaze target positions associated with the identified potential gaze target, across multiple video frames received at block 402 have. In some embodiments, the computing device 300 may select a potential gaze target having a velocity that substantially matches the detected gaze target velocity, including, for example, a substantially matching velocity and direction. In some embodiments, the computing device 300 may select a potential gaze target having an acceleration that substantially matches the detected gaze target acceleration, including, for example, an acceleration of substantially matching magnitude and direction . In some embodiments, the computing device 300 may dynamically assign weights to a plurality of potential gaze targets identified in block 404, as described herein, and the computing device 300 may determine the potential gaze Dynamically assigned weights may be used to select the target. Block 405 may be followed by block 406.

At block 406, the computing device 300 may determine, at block 405, the position of the selected gaze target, the velocity, and / As such, it may change the detected gaze target, including its positions, velocities, and / or accelerations. In some embodiments, the modified gaze target information may be stored or transmitted to another computer for storage and / or further analysis. In some embodiments, information describing the potential gaze target selected at block 405 may be stored or transmitted with the modified gaze target information, e.g., a snapshot of the selected gaze target, The text identifying the advertisement in the selected line-of-sight target may be stored or transmitted along with the changed line-of-sight target information. Block 406 may be followed by block 407.

In the "Apply Changed Gaze Target Information" block 407, the computing device 300 may apply the modified gaze target information from block 406 with respect to any application where the ocular tracking information may be used. For example, the computing device 300 may determine the AR information for the display, such as, by way of example, identifying the modified object from block 406 to identify, describe, or otherwise determine the object The gaze target information may also be used. In another example, the computing device 300 may apply the modified gaze target information from block 406 as a user input for controlling at least one computing device function. As will be appreciated, a wide variety of other applications are possible, and the disclosure is not limited to any particular application.

In some embodiments, blocks 401 through 406, and optionally also block 407, may be performed by computing device 300 in real time, i. E., At block 401 and block 402, The information may include real-time visual direction information and real-time digital scene information, respectively. The computing device 300 may further perform blocks 403 through 407 in real time when real-time gaze direction information and real-time digital scene information are received.

5 is a block diagram illustrating an exemplary eye tracking accuracy enhancer arranged in accordance with at least some embodiments of the present disclosure. FIG. 5 illustrates an eye tracking accuracy enhancer 500 that may serve as an alternative arrangement for the eye tracking accuracy enhancer 323 as described in connection with FIGS. 3 and 4. FIG. The eye tracking accuracy enhancer 500 includes a potential gaze target detector 510, a potential gaze target selector 520, a velocity extractor 530 and a velocity extractor 540, an acceleration extractor 550, a filter 560, An extractor 570, and an ocular machine model simulator 580. The potential gaze target selector 520 includes a weighted position comparator 521, a weighted velocity comparator 522, and a weighted acceleration comparator 523.

5, the potential gaze target selector 520 may receive the gaze coordinate stream 501 as input and the gaze coordinate stream 501 may include the output from the eye-to-op sensor 103, Or may be derived from its output. Potential gaze target selector 520 may also receive potential gaze target coordinate streams 502 as input and potential gaze target coordinate streams 502 may identify potential gaze targets and provide corresponding potential May include an output from a potential gaze target detector 510 that analyzes the digital scene information from the digital camera 102 to extract in-line target coordinate streams 502, It is possible. The potential gaze target selector 520 may generate the selected gaze target coordinate stream 503 as an output and the gaze tracking accuracy enhancer 500 may add the selected gaze target coordinate stream 503 to the modified gaze coordinate stream 504, As shown in Fig.

In some embodiments, the potential gaze target detector 510 may analyze the video stream received from the digital camera 102 and extract potential gaze targets therefrom. The potential gaze target detector 510 may provide potential gaze target coordinate streams 502, including a stream of position coordinates for each identified potential gaze target. The eye tracking accuracy enhancer 500 may optionally convert the potential gaze target coordinate streams 502 into any desired coordinate space, as described herein. On the other hand, the eye tracking accuracy enhancer 500 may also convert the eye coordinate stream 501 into the coordinate space that is coincident with the coordinate space of the potential eye target coordinate streams 502. Eye-gaze coordinate stream 501 may include a stream of position coordinates for the eye being tracked.

In some embodiments, the weighted position comparator 521 may compare the position coordinates in each of the potential gaze target coordinate streams 502 with the position coordinates in the gaze coordinate stream 501. The weighted position comparator 521 may assign a weight to each identified potential gaze target based on its proximity to the position coordinates in the gaze coordinate stream 501 of its corresponding potential gaze target coordinate stream . Weighted position comparator 521 may assign stronger weights to identified potential gaze targets having coordinate streams that are closer in closer to eye gaze coordinate stream 501. [ In some embodiments, the gaze coordinate stream 501 and the potential gaze target coordinate streams 502 may include time information, and the weighted position comparator 521 may include position coordinates at a plurality of different time points May be adapted to compare. In some embodiments, the weighted position comparator 521 may be adapted to remove identified potential gaze targets having coordinate streams substantially including position coordinates outside the error bounds for the gaze coordinate stream 501. [ The potential gaze targets removed may temporarily disappear as candidates for the current selection action by the eye tracking accuracy enhancer 500. [

In some embodiments, the velocity extractor 530 may be adapted to compute a linear time derivative d / dt of the gaze coordinate stream 501, thereby computing the gaze velocity. Likewise, the velocity extractor 540 may calculate the one-time derivatives d / dt of each of the potential gaze target coordinate streams 502 so that the weighted position comparator 521 may determine May be adapted to compute the speed for each identified potential gaze target rather than the identified potential gaze targets.

The weighted velocity comparator 522 may be adapted to compare the calculated velocities of each of the potential gaze target coordinate streams 502 with the calculated velocity of the gaze coordinate stream 501. The weighted velocity comparator 522 may assign a weight to each identified potential gaze target based on the degree of match between the calculated velocity and the calculated velocity of the gaze coordinate stream 501. [ Weighted velocity comparator 522 may assign stronger weights to identified potential eye targets having velocities that more closely match velocity of eye coordinate stream 501. [ In some embodiments, the weighted velocity comparator 522 may apply generally stronger weights to identified potential eye targets having a velocity that substantially matches the eye velocity of the eye coordinate stream 501, for example, Weighted position comparator 521 may assign more weighting values than the weights applied by the weighted position comparator 521 while the weighted velocity comparator 522 may otherwise assign a velocity that does not substantially match the velocity of the eye coordinate stream 501 It is also possible to assign a small or no weight to the identified potential gaze targets. The weighted speed comparator 522 may be adapted to compare speeds at a number of different points in time. In some embodiments, the weighted velocity comparator 522 may be configured to calculate velocities that are substantially different from the velocities calculated for the eye coordinate stream 501, such as, for example, substantially different directions, , Or substantially different sizes, such as, for example, less than or equal to 50% of the gaze speed, or less than or equal to 200% of the gaze speed.

In some embodiments, the acceleration extractor 550 may be adapted to compute the line-of-sight acceleration by calculating the time-of-day derivative d ² / dt ² of the line of sight coordinate stream 501. Similarly, the acceleration extractor 570 is a potential eye target coordinate streams 502, each of the Second time derivative d ² / dt ² by calculating, a weighted position comparator 521 and / or weighting speed comparator (522 May be adapted to compute the acceleration for each identified potential gaze target, rather than the identified potential gaze targets that may be removed from the current selection operation by the user.

The filter 560 may be adapted to filter the calculated acceleration outputs from the acceleration extractor 550. In some cases, the acceleration outputs may prove to be too noisy for useful comparisons. The filter 560 may be used to remove, for example, certain fast-back-and-forth type eye accelerations and / or to remove " rapidly changing " small- and short- The acceleration outputs may be smoothed to capture a wider range of eye acceleration information.

Eye model simulator 580 is adapted to modify the calculated accelerations for each of the potential gaze target coordinate streams 502 by simulating eye accelerations for a hypothetical eye following the corresponding identified potential gaze targets It is possible. In some embodiments, the ocular machine model simulator 580 may add, for example, lag and overshoot motion to the calculated accelerations for each of the potential gaze target coordinate streams 502. In some embodiments, the ocular machine model simulator may be implemented in place of, or in addition to, the ocular mechanical model simulator 580 between the acceleration extractor 570 and the weighted acceleration comparator 523, And may be disposed between the comparator 522 and / or between the potential gaze target coordinate streams 502 and the weighted position comparator 521.

The weighted acceleration comparator 523 optionally changes the calculated accelerations of each of the potential gaze target coordinate streams 502, as optionally modified by the ocular machine model simulator 580, May be adapted to compare with the calculated acceleration of gaze coordinate stream 501, as shown. The weighted acceleration comparator 523 may assign a weight to each identified potential gaze target based on the degree of match between the calculated acceleration and the calculated acceleration of the gaze coordinate stream 501. [ The weighted acceleration comparator 523 may assign stronger weights to identified potential eye targets having an acceleration that more closely matches the acceleration of the eye coordinate stream 501. [ In some embodiments, the weighted acceleration comparator 523 may apply very strong weights to identified potential eye targets having accelerations that substantially match the acceleration of the eye coordinate stream 501, for example, Weighted acceleration comparator 523 may assign more weighting values than the weights applied by the position comparator 521 and / or the weighted velocity comparator 522, while the weighted acceleration comparator 523 may, And may assign a small or no weight to the identified potential gaze targets having substantially non-matching accelerations. The weighted acceleration comparator 523 may be adapted to compare accelerations at a number of different time points.

In some embodiments, the potential gaze target selector 520 may include a weighted position comparator 521, a weighted velocity comparator 522, and / or a weighted acceleration comparator 522 for each identified potential gaze target 523, < / RTI > The potential gaze target selector 520 may select an identified potential gaze target with the strongest aggregation weight. The potential gaze target selector 520 may output a selected gaze target coordinate stream 503, including, for example, a coordinate stream among the potential gaze target coordinate streams 502 corresponding to the selected gaze target. In some embodiments, the selected line of sight target coordinate stream 503 is selected such that different identified potential line of sight targets are dynamically selected by the potential line of sight target selector 520 based on changes in the line of sight coordinate stream 501 In real time, dynamically switch between different selected line of sight target coordinate streams. The eye tracking accuracy enhancer 500 may store, transmit, or otherwise apply the selected eye target coordinate stream 503 as a modified eye coordinate stream 504.

6 is a diagram illustrating a scene including potential gaze targets, detected gaze targets, selected gaze targets, and AR information for a selected gaze target, arranged in accordance with at least some embodiments of the present disclosure. Figure 6 illustrates a scene 600. Scene 600 includes a potential gaze target 601 showing motion M (F1), a potential gaze target 602 showing motion M (F2), and a potential gaze target 603 showing motion M (F3) . The detected sight line target 141 may show motion M (DT). The potential gaze targets 601 through 603, the detected gaze target 141, and their motions may be obtained as disclosed herein, and the potential gaze target 603 may be obtained from techniques described herein And thus may be selected as the selected gaze target 142.

In FIG. 6, a computing device, such as computing device 300, may determine AR information 610 for a selected gaze target 142. For example, when a user of the computing device 300 is viewing a car race, the computing device 300 may compare the AR information identifying the car at the selected gaze target 142, optionally with the speed, Operator, or other information. This AR information may be retrieved from the information stored locally at computing device 300 and / or from the computer network using information derived from scene 600 to identify the vehicle at the selected gaze target 142 .

7 is a diagram illustrating a scene including potential gaze targets, a detected gaze target, and a selected gaze target, wherein the selected gaze target is arranged in accordance with at least some of the embodiments of the disclosure, And may be applied as a user input for controlling the device function. Figure 7 illustrates a scene 700. The scene 700 includes a visual line target 701, a potential gaze target 702, a potential gaze target 703, a potential gaze target 704, a potential gaze target 705, 706, and a potential gaze target 707, and the potential gaze targets 701 through 705 are within the UI displayed on the computing device display. Potential gaze targets 706 and 707 include other potential gaze targets identified in scene 700, for example. The potential gaze targets 701 through 707 and the detected gaze target 141 may be obtained as disclosed herein and the potential gaze target 703 may be obtained from the selected gaze target May be selected as the target 142. Eye tracking matching may be useful for matching eye motion to the motion of the view of still objects, even as the user head motion may cause the frozen objects to change their position in the ocular coordinate system.

7, a computing device, such as computing device 300, may apply a selected gaze target 142 as a user input for controlling computing device functions, e.g., the functionality of a computing device in scene 700, have. The computing device 300 may send a command to the computing device in the scene 700, for example, to select a UI element at the potential gaze target 703. The computing device 300 recognizes the UI 710 and its potential gaze targets as well as the computing device in the scene 700 so that the computing device 300 can determine appropriate UI cursors for the computing device in the scene 700 To be transmitted. Alternatively, the computing device 300 may be configured to recognize the UI 710 and its various controls in advance and communicate with the computing device in the scene 700, for example, by a user. In some embodiments, the computing device 300 may replicate the functions of the UI 710 at the computing device 300 and optionally control the functions of the computing device 300. [

There is little difference between the hardware and software implementations of the aspects of the systems; The use of hardware or software is typically a design choice that represents cost-to-efficiency tradeoffs (but in some contexts, the choice between hardware and software may not always be important). There are a variety of means (e.g., hardware, software, and / or firmware) that may enable the processes and / or systems and / or other techniques described herein, And / or systems and / or other techniques may vary depending on the context in which they are used effectively. For example, if the implementer determines that speed and accuracy are more important than others, the implementer may chose primarily hardware and / or firmware means; If the flexibility is more important than anything else, the implementer may chose mainly the software implementation; Alternatively, the implementer may select some combination of hardware, software, and / or firmware.

The foregoing detailed description sets forth various embodiments of devices and / or processes through the use of block diagrams, flowcharts, and / or examples. Insofar as such block diagrams, flowcharts, and / or examples include one or more functions and / or operations, it is contemplated that each function and / or operation within such block diagrams, flowcharts, And / or collectively, by various hardware, software, firmware, or virtually any combination thereof, as will be understood by those skilled in the art. In one embodiment, various portions of the subject matter described herein may be implemented as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), or other integrated formats ≪ / RTI > However, those skilled in the art will appreciate that certain aspects of the embodiments disclosed herein may be wholly or partially implemented in integrated circuits, as one or more computer programs (e.g., executed on one or more computer systems) (E. G., As one or more programs running on one or more microprocessors), firmware, or virtually any combination thereof, as well as one or more programs running on one or more processors And that the design of the circuitry and / or the writing of software and / or code for the firmware will be within the skill of those skilled in the art in light of the present disclosure. It will also be appreciated by those skilled in the art that the mechanisms of the subject matter described herein may be distributed as various types of program products and that the exemplary embodiments of the subject matter described herein may be applied to any particular type of signal bearing medium Regardless of the type. Examples of signal bearing media include, but are not limited to, the following: recordable type media such as floppy disks, hard disk drives, compact discs (CD), digital video discs (DVD), digital tapes, And transmission-type media, such as digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.).

Those skilled in the art will recognize that it is common in the art to describe devices and / or processes in the manner described herein, and then utilize engineering practice to integrate these described devices and / or processes into data processing systems . That is, at least some of the devices and / or processes described herein may be incorporated into a data processing system through a reasonable amount of experimentation. Those skilled in the art will appreciate that a typical data processing system generally includes a plurality of processors, such as system unit housings, video display devices, memories such as volatile and non-volatile memory, microprocessors and digital signal processors, operating systems, drivers, (E. G., Position and / or position) and / or control systems (e. G., Position and / or orientation) including computation entities such as, for example, / RTI > and / or feedback for sensing velocity; and control motors for moving and / or adjusting components and / or quantities). A typical data processing system may be implemented utilizing any suitable commercially available components such as those commonly found in data computing / communications and / or network computing / communication systems. The principles described herein sometimes illustrate different components that are included in or associated with different other components. It will be appreciated that these depicted architectures are merely examples and in fact many different architectures may be implemented that achieve the same functionality. In the conceptual sense, any arrangement of components to achieve the same functionality is effectively " related " to achieve the desired functionality. Thus, any two components that are combined herein to achieve a particular functionality may appear to be " related " to each other such that the desired functionality is achieved, regardless of the architectures or intermediate components. Likewise, any two components so associated may also appear to be " operably coupled " or " operatively coupled " to one another to achieve the desired functionality, and any two components And may also appear " operably coupled " to one another to achieve the desired functionality. Specific examples of operably coupleable include components that are physically connectable and / or physically interacting and / or components that interact wirelessly and / or wirelessly and / or logically interact ≪ / RTI > and / or logically interactable components.

In connection with the use of substantially any plural and / or singular terms herein, those skilled in the art will be able to translate from plural to singular and / or singular plural as appropriate to the context and / or application. The various singular / plural substitutions may be explicitly recited herein for clarity.

In general, terms used herein and in particular in the appended claims (e.g., in the text of the appended claims) generally refer to terms such as "open" terms (e.g., Quot; is to be construed as " including but not limited to, " and the term " having " should be interpreted as having at least, and the term " comprising " It will be understood. It will also be appreciated by those skilled in the art that, if a particular number of the recited claims is intended, such intent will be expressly set forth in the claims, and such intent is not present in the absence of such description. For example, the following appended claims may include the use of " at least one " and " more than one " However, the use of such phrases is not intended to limit the scope of the present invention to any and all claims, including the introduction of claim statements by " a " or " an "Quot; an " and " an " should be construed to mean " at least one " or " Should not be construed to mean that the invention is limited to inventions that include only one such disclosure; The same is true in the case of the use of articles used to introduce claims. In addition, those skilled in the art will recognize that such descriptions should be construed to mean at least the stated number (s), even if a specific number of the introduced claims is explicitly stated (for example, "Quot; base " means at least two bases, or two or more bases). Moreover, in those instances where an analogous provision is used, such as " at least one of A, B, and C ", this structure is generally intended to mean that one of ordinary skill in the art will understand the promise A, B, and C together, A and C together, B and C together, and / or A, B, and C together, etc. But are not limited to, systems with a < / RTI > In those cases where an appointment similar to " at least one of A, B, or C " is used, such structure is generally intended to mean that one of ordinary skill in the art will understand the promise (e.g., " A, B, A and B together, A and C together, B and C together, and / or A, B, and C together and so on But are not limited to, systems). In addition, virtually any declarative word and / or phrase that suggests two or more alternative terms may be used in either the detailed description, the claims, or the drawings, either as terms, either one of the terms, It will be understood by those skilled in the art that the present invention should be understood as taking into account the possibilities to include. For example, the phrase "A or B" will be understood to include the possibilities of "A" or "B" or "A and B".

Although certain exemplary techniques have been described and shown herein using various methods, devices and systems, it will be appreciated by those skilled in the art that various other changes may be made and equivalents may be substituted without departing from the claimed subject matter Should be understood. In addition, many modifications may be made to adapt a particular situation to the teachings of the subject matter claimed without departing from the central concept set forth herein. It is therefore intended that the claimed subject matter is not limited to the specific examples disclosed, but that such claimed subject matter may also include all implementations falling within the scope of the claimed claims, and equivalents thereof.

Claims (23)

As an eye tracking method,
Receiving, by the computing device, gaze direction information for at least one eye from an eye-facing sensor;
Receiving, by the computing device, digital scene information from a digital camera, the digital camera being oriented to capture the digital scene information from a scene viewed by the at least one eyeball;
Analyzing the digital scene information by the computing device to identify one or more potential gaze targets within the digital scene information;
Translating, by the computing device, the positions of the one or more potential gaze targets into estimated gaze directions; And
Wherein the viewing direction information from the eye-facing sensor is estimated by the computing device to be an estimated one of the one or more potential gaze targets that corresponds to the positions of the one or more potential gaze targets Lt; RTI ID = 0.0 > direction, < / RTI >
Wherein the estimated eye direction corresponding to the selected eye target is within an error boundary surrounding the eye direction information received from the eye-facing sensor. The method according to claim 1,
Wherein the gaze direction information comprises angular coordinates. The method according to claim 1,
Analyzing the digital scene information to identify one or more potential gaze targets within the digital scene information by the computing device comprises performing object recognition within the digital scene information to identify potential And determining in-sight targets. The method according to claim 1,
Analyzing the digital scene information to identify one or more potential gaze targets within the digital scene information by the computing device comprises identifying one or more sets of related graphical features within the digital scene information, ≪ / RTI > comprising determining the potential gaze targets including the set of gaze targets. The method according to claim 1,
Wherein the one or more potential gaze targets are:
A stationary gaze target in the digital scene information;
A moving gaze target in the digital scene information; or
An accelerating gaze target in the digital scene information;
≪ / RTI > 6. The method of claim 5,
Modeling eye movements associated with following the moving gaze target or the accelerating gaze target in the physical scene by the computing device; And
Further comprising comparing, by the computing device, modeled eye motions with eye motion detected in the eye-facing sensor, to select the eye gaze target from the one or more identified potential eye gaze targets. Way. The method according to claim 6,
Wherein modeling eye movements associated with following the moving eye target or the accelerating eye target in the physical scene comprises applying an eye mechanical model simulator. The method according to claim 1,
Selecting the gaze target from among the one or more potential gaze targets identified by the computing device comprises dynamically assigning weights to the plurality of potential gaze targets, ≪ / RTI > using the weights assigned to the eye. The method according to claim 1,
Further comprising, by the computing device, determining augmented reality information for display based on the selected line of sight target. The method according to claim 1,
Further comprising, by the computing device, applying the selected gaze target as a user input for controlling at least one computing device function. The method according to claim 1,
Wherein the digital camera has at least one of a default frame rate or a default resolution,
The eye tracking method may further comprise, by the computing device, operating the digital camera at one or more of a reduced frame rate, a reduced resolution, an improved frame rate, or an improved resolution to further capture the digital scene information ≪ / RTI &
Wherein the reduced frame rate is lower than the default frame rate, the reduced resolution is lower than the default resolution, the improved frame rate is higher than the default frame rate, and the improved resolution is less than the default resolution High, eye tracking method. The method according to claim 1,
Wherein the gaze direction information and the digital scene information each include real-time gaze direction information and real-time digital scene information,
Wherein the eye tracking method further comprises changing the selected line-of-sight targets in real-time when the real-time line-of-sight direction information and the real-time digital scene information are received. As an eye tracking computing device,
A processor;
Memory; And
An eye tracker stored in the memory and executable by the processor, the eye tracker comprising:
Receiving eye-gaze direction information for at least one eye from an eye-facing sensor;
Receiving digital scene information from a digital camera, the digital camera being adapted to capture the digital scene information from a scene viewed by the at least one eyeball;
Analyze the digital scene information to identify one or more potential gaze targets within the digital scene information;
Transforming the positions of the one or more potential gaze targets into estimated gaze directions; And
Comparing the gaze direction information from the eye-facing sensor with the estimated gaze directions corresponding to the positions of the one or more potential gaze targets, to select a gaze target from among the identified one or more potential gaze targets Lt; / RTI >
Wherein the estimated eye direction corresponding to the selected eye target is within an error boundary surrounding the eye direction information received from the eye-facing sensor. 14. The method of claim 13,
Wherein the gaze direction information comprises angular coordinates. 14. The method of claim 13,
Wherein the one or more potential gaze targets are:
A stationary gaze target in the digital scene information;
A moving gaze target in the digital scene information; or
An accelerating gaze target in the digital scene information;
/ RTI > The device of claim 1, 16. The method of claim 15,
The eye tracker comprises:
Model eye movements associated with following the moving gaze target or the accelerating gaze target in the physical scene; And
And compare the modeled eye motions with the eye motion detected in the eye-facing sensor to select the eye gaze target from among the one or more potential eye gaze targets identified. 17. The method of claim 16,
The eye tracker comprises:
Is configured to model eye movements associated with following the moving gaze target or the accelerating gaze target at least partially within the physical scene by applying an eye mechanical model simulator, device. 14. The method of claim 13,
The eye tracker comprises:
And determine augmented reality information for display based on the selected line of sight target. 14. The method of claim 13,
The eye tracker comprises:
And to apply the selected gaze target as a user input for controlling at least one computing device function. 14. The method of claim 13,
Wherein the gaze direction information and the digital scene information each include real-time gaze direction information and real-time digital scene information,
Wherein the eye tracker is configured to change selected line-of-sight targets in real-time when the real-time gaze direction information and the real-time digital scene information are received. 20. A non-transitory computer readable storage medium having computer-executable instructions executable by a processor,
Wherein the instructions, when executed by the processor, cause the processor to:
Receiving eye-facing information from the eye-facing sensor for at least one eye;
Receiving digital scene information from a digital camera, the digital camera being adapted to capture the digital scene information from a scene viewed by the at least one eyeball;
Analyze the digital scene information to identify one or more potential gaze targets within the digital scene information;
Convert the positions of the one or more potential gaze targets into estimated gaze directions; And
Comparing the gaze direction information from the eye-facing sensor with the estimated gaze directions corresponding to the positions of the one or more potential gaze targets, to select a gaze target from among the identified one or more potential gaze targets Non-volatile computer-readable storage medium. 22. The method of claim 21,
Wherein the estimated viewing direction corresponding to the selected gaze target is within an error boundary surrounding the gaze direction information received from the eye-facing sensor. delete

Images