US20150062323A1 - Portable eye tracking device - Google Patents

Portable eye tracking device Download PDF

Info

Publication number
US20150062323A1
US20150062323A1 US14281676 US201414281676A US2015062323A1 US 20150062323 A1 US20150062323 A1 US 20150062323A1 US 14281676 US14281676 US 14281676 US 201414281676 A US201414281676 A US 201414281676A US 2015062323 A1 US2015062323 A1 US 2015062323A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
user
image data
eye
portion
illuminator
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US14281676
Other versions
US9710058B2 (en )
Inventor
Simon Gustafsson
Jonas Andersson
Erik Engwall
Richard HAINZL
Lars-Ola Leksén
Markus Cederlund
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tobii AB
TOBBI TECHNOLOGY AB
Original Assignee
TOBBI TECHNOLOGY AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/2251Constructional details
    • H04N5/2254Mounting of optical parts, e.g. lenses, shutters, filters or optical parts peculiar to the presence or use of an electronic image sensor
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/113Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/01Head-up displays
    • G02B27/017Head mounted
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013Eye tracking input arrangements
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/70Determining position or orientation of objects or cameras
    • G06T7/73Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/2256Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles provided with illuminating means
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R1/00Details of transducers, loudspeakers or microphones
    • H04R1/20Arrangements for obtaining desired frequency or directional characteristics
    • H04R1/32Arrangements for obtaining desired frequency or directional characteristics for obtaining desired directional characteristic only
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/01Head-up displays
    • G02B27/0101Head-up displays characterised by optical features
    • G02B2027/0138Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/01Head-up displays
    • G02B27/0101Head-up displays characterised by optical features
    • G02B2027/014Head-up displays characterised by optical features comprising information/image processing systems
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/01Head-up displays
    • G02B27/017Head mounted
    • G02B2027/0178Eyeglass type, eyeglass details G02C
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS, OR APPARATUS
    • G02B27/00Other optical systems; Other optical apparatus
    • G02B27/01Head-up displays
    • G02B27/0179Display position adjusting means not related to the information to be displayed
    • G02B2027/0187Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10141Special mode during image acquisition
    • G06T2207/10152Varying illumination
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30196Human being; Person
    • G06T2207/30201Face
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2201/00Details of transducers, loudspeakers or microphones covered by H04R1/00 but not provided for in any of its subgroups
    • H04R2201/40Details of arrangements for obtaining desired directional characteristic by combining a number of identical transducers covered by H04R1/40 but not provided for in any of its subgroups

Abstract

A portable eye tracker device is disclosed which includes a frame, at least one optics holding member, and a control unit. The frame may be adapted for wearing by a user. The at least one optics holding member may include at least one illuminator configured to selectively illuminate at least a portion of at least one eye of the user, and at least one image sensor configured to capture image data representing images of at least a portion of at least one eye of the user. The control unit may be configured to control the at least one illuminator for the selective illumination of at least a portion of at least one eye of the user, receive the image data from the at least one image sensor, and calibrate at least one illuminator, at least one image sensor, or an algorithm of the control unit.

Description

    CROSS-REFERENCE TO RELATED APPLICATIONS
  • This application claims priority to Provisional U.S. Patent Application No. 61/873,154 filed Sep. 3, 2013, entitled “PORTABLE EYE TRACKING DEVICE,” the entire disclosure of which is hereby incorporated by reference, for all purposes, as if fully set forth herein.
  • This application is also related to U.S. patent application Ser. No. 14/281,587 filed concurrently herewith, entitled “PORTABLE EYE TRACKING DEVICE,” U.S. patent application Ser. No. 14/281,599 filed concurrently herewith, entitled “PORTABLE EYE TRACKING DEVICE,” and U.S. patent application Ser. No. 14/281,616 filed concurrently herewith, entitled “PORTABLE EYE TRACKING DEVICE,” the entire disclosures of which are hereby incorporated by reference, for all purposes, as if fully set forth herein.
  • BACKGROUND OF THE INVENTION
  • Methods for determining a point at which a user is looking are known in the art. The technology is often referred to as eye tracking or gaze detection technology. Eye tracking systems are typically found in two major forms. In one form, a sensor or other eye tracking device is located remote from a user's eye such as in or on a computer, display or similar device.
  • One known method of eye tracking includes the use of infrared light and an image sensor. The infrared light is directed towards the pupil of a user and the reflection of the light is captured by an image sensor. Through analysis of the reflection point, the direction of the user's gaze may be calculated. One such system is described in U.S. Pat. No. 7,572,008 assigned to Tobii Technology AB, Sweden (the “'008 patent”). The entire disclosure of the '008 patent is hereby incorporated by reference, for all purposes, as if fully set forth herein.
  • Portable or wearable eye tracking devices have also been previously described and are available for commercial purchase. One such eye tracking system is described in U.S. Patent Application Publication Number 2011/0279666 assigned to Tobii Technology AB, Sweden (the “'666 Application”). The entire disclosure of the '666 Application is hereby incorporated by reference, for all purposes, as if fully set forth herein. The '666 Application describes a wearable eye tracking device that requires an external infrared light source to be placed in a scene as a reference point, to assist in determining the direction of a user's gaze.
  • Existing portable eye tracking systems may suffer severe performance degradation when the equipment moves relative to the wearer's head. For example, glasses may slip relative to a wearer's nose; in addition, a wearer may manually adjust glasses as they are worn. For designs requiring calibration, such movement of the glasses relative to the wearer's head may negate the calibration and significantly degrade the accuracy of the readings. As another example, a single-camera portable eye tracking system may render substantially degraded reading in certain conditions including when the wearer is in the presence of strong light sources; is exposed to direct sunlight; or when the single camera's view is obstructed such as by an eyelash. Furthermore, such single-camera systems may be unable to detect gaze directions at the extremities of a user's field of view.
  • BRIEF SUMMARY OF THE INVENTION
  • In one embodiment, a portable eye tracker device is provided. The portable eye tracer device may include a frame, at least one optics holding member, and a control unit. The frame may be adapted for wearing by a user. The at least one optics holding member may include at least one illuminator configured to selectively illuminate at least a portion of at least one eye of the user, and at least one image sensor configured to capture image data representing images of at least a portion of at least one eye of the user. The control unit may be configured to control the at least one illuminator for the selective illumination of at least a portion of at least one eye of the user, receive the image data from the at least one image sensor, and calibrate at least one illuminator, at least one image sensor, or an algorithm of the control unit.
  • In another embodiment, a method of determining a gaze direction for a user is provided. The method may include activating at least one illuminator on a frame worn by a user to selectively illuminate at least a portion of at least one of the user's eyes. The method may also include receiving image data representing images at least a portion of at least one of the user's eyes from at least one image sensor on the frame. The method may further include calibrating at least one illuminators, at least one image sensor, or an algorithm of an associated control unit. The method may additionally include determining a gaze target area for the user based at least in part on the image data.
  • In another embodiment, a non-transitory machine readable medium having instructions thereon for determining a gaze direction for a user is provided. The instructions may be executable by a processor for activating at least one illuminator on a frame worn by a user to selectively illuminate at least a portion of at least one of the user's eyes. The instruction may also be executable for receiving image data representing images at least a portion of at least one of the user's eyes from at least one image sensor on the frame. The instructions may further be executable for calibrating at least one illuminators, at least one image sensor, or an algorithm of an associated control unit. The instructions may additionally be executable for determining a gaze target area for the user based at least in part on the image data.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • The present invention is described in conjunction with the appended figures:
  • FIG. 1A shows an eye tracking device according to some embodiments of the present invention;
  • FIG. 1B shows an alternative eye tracking device according to other embodiments of the present invention;
  • FIG. 2A shows a rear view of an eye tracking device according to some embodiments of the present invention;
  • FIG. 2B shows a rear view of an eye tracking device according to other embodiments of the present invention;
  • FIG. 3 shows a first view of an optics holding member according to some embodiments of the present invention;
  • FIG. 4 shows a second view of an optics holding member according to some embodiments of the present invention;
  • FIG. 5 shows one possible method employed by various embodiments of the invention; and
  • FIG. 6 is a block diagram of an exemplary computer system capable of being used in at least some portion of the apparatuses or systems of the present invention, or implementing at least some portion of the methods of the present invention.
  • In the appended figures, similar components and/or features may have the same numerical reference label. Further, various components of the same type may be distinguished by following the reference label by a letter that distinguishes among the similar components and/or features. If only the first numerical reference label is used in the specification, the description is applicable to any one of the similar components and/or features having the same first numerical reference label irrespective of the letter suffix.
  • DETAILED DESCRIPTION OF THE INVENTION
  • The ensuing description provides exemplary embodiments only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the ensuing description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. It shall be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the appended claims. For example, any detail of one embodiment discussed herein may or may not be present in all possible variations of that embodiment, and may or may not be present in all possible variations of other embodiments discussed herein.
  • Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other elements in the invention may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.
  • Also, it is noted that individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process may be terminated when its operations are completed, but could have additional steps not discussed or included in a figure. Furthermore, not all operations in any particularly described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.
  • The term “machine-readable medium” includes, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels and various other mediums capable of storing, containing or carrying instruction(s) and/or data. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.
  • Furthermore, embodiments of the invention may be implemented, at least in part, either manually or automatically. Manual or automatic implementations may be executed, or at least assisted, through the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine readable medium. A processor(s) may perform the necessary tasks.
  • Referring now to FIG. 1A a portable eye tracking device 10 according to one embodiment of the invention is shown. Eye tracking device 10 includes a frame 12 having a glasses lens (or pair of lenses) 14 coupled with it in such a manner so as to resemble a traditional pair of eye glasses or sunglasses. Frame 12 may include a center piece 15 to which glasses lens 14 is attached and from which two arms 17 (i.e., temples) extend. Arms 17 may fit above the ear of a user in the manner of a traditional pair of eye glasses or sun glasses.
  • In some embodiments, a nose piece 16 may be provided for comfort and to assist in fitting of device 10 to a user's nose. A scene camera 18 (which may capture still and/or video images and/or other image data) may be disposed in the middle of glass lens 14, or between separate glass lenses, above the nose piece 16. A microphone 20 may also be placed adjacent or near to scene camera 18.
  • A control unit 22 may be located within one or both arms 17, either in the front, middle, and/or end thereof. Any processor function described herein may be executed at control unit 22, and/or an external processor in communication with control unit 22. Control unit 22 may include elements for performing computational tasks, such as a printed circuit board (PCB) and other electronics, as will be described in further detail herein. Control unit 22 may also contain a communications port or component designed to communicate with an external computing device. This communications port or device may support any one or more form of known communication. For example, a communications port may include a Universal Serial Bus (USB) port, Firewire port, High-Definition Multimedia Interface (HDMI) port, Ethernet port or the like. A communications device may include a Wi-Fi transceiver, Bluetooth transceiver, or any other near-field or longer-range communication device. In other embodiments, the communications port or device may also be of a proprietary type especially designed for use in a portable eye tracking device. The communication port or device may for example include a low-powered wireless communications means.
  • FIG. 1B, shows an alternative eye tracking device 10A with certain components in different positions. In this embodiment, control unit 22 may be located on the side of arm 17, and include an HDMI interface 19.
  • FIG. 2A shows eye tracking device 10 according to certain embodiments of the invention as viewed from behind. For illustration purposes a portion of center piece 12 has been removed from the figure to show the placement of a circuit board 50 which may contain electronic components such as flex connectors 52 and microphone 20. Circuit board 50 may also contain processing architecture such as a digital signal processor, field programmable gate array and/or another processor system on a chip. Circuit board 50 may be attached to center piece 12 by way of a conventional fastening means, such as screws 54, adhesive, and/or other means. Center piece 12 may include one or more parts which fit over circuit board 50, such that circuit board 50 is disposed within center piece 12, and not visible during normal use.
  • Device 10 may also include two optics holding members 70. In other embodiment only one contiguous optics holding member may be provided which provides the functionality of the two optics holding members 70 shown in FIG. 2A. Each optics holding member 70 may include one or more image sensors and one or more illuminators as will be further explained below. Thus, an optics holding member 70 may include components that illuminate a user's eye, or some sub-portion thereof (for example, the iris), and capture images comprising reflection points of the illumination on the user's eye, or some sub-portion thereof. FIG. 2B, shows the alternative eye tracking device 10A of FIG. 1B.
  • FIGS. 3 & 4 show optics holding members 70 according to embodiments of the invention in further detail. Each optics holding member 70 may include two primary components, a cover 72 and a flexible member 74. Cover 72 may cover flexible member 74 when properly coupled together as shown by optics holding member 70B. Optics holding member 70A is shown with flexible member 74 and cover 72 parted in order to show the interaction between the two components. Flexible member 74 and cover 72 may be configured such that the flexible member 74 snaps into the cover 72. As another example, the flexible member 74 and cover 72 may be connected to one another using adhesive, screws and/or other conventional fastening means.
  • Flexible member 74 may include a flex circuit and have mounted on it one or more spaced illuminators 76 and/or one or more image sensors 78. Any number of illuminators 76 (e.g., 1, 2, 3, 4, 5, 6, etc.) and image sensors (e.g., 1, 2, 3, 4, 5, 6, etc.) may be employed in each optics holding member 70. Illuminator 76 may include infrared illuminators capable of emitting infrared light as would be readily understood by a person of skill in the art, but other illuminators may also be used, including illuminators emitting ultraviolet and/or visible light. Image sensor 78 may be sensitive to the light emitted by illuminator 76 as well as visible light and other light that may be present in the area of use. For example, when device 10 is used outside, ambient light in the visible and nonvisible spectrums (e.g., visible and ultraviolet light from the sun) may be present and image sensor 78 may be configured to detect and distinguish (e.g., filter out or otherwise compensate for) different wavelengths of ambient light. Flexible member 74 may include a power source (e.g. batteries, solar cell, etc.) to provide power to illuminator 76 and image sensor 78 in a manner that would be well understood by a person of normal skill in the art. An upper end of the flexible member 74 may include a contact element 80 that connects to circuit board 50 mounted on center piece 12. The use of flexible circuits and electronics is well known, and a person skilled in the art would appreciate the way they can be used in the present invention.
  • In some embodiments of the invention, cover 72 may be formed from an optically transparent material such that light from illuminators 76 may pass through cover 72 substantially unhindered. In some embodiments, cover 72 may be shaped to transmit the light emitted by illuminators 76 to the appropriate location. Parts or areas of cover 72 may for example be shaped as a lens that diverges light from one or more of illuminators 76 (or even ambient lighting sources) to illuminate a larger area including the user's eyes. Parts or areas of cover 72 may also be shaped to converge or focus light on particular areas or locations. Cover 72 may be molded out of a single piece of material wherein areas of the material are shaped like lenses or otherwise shaped to transmit light as described above. In other embodiments the cover may include multiple parts that are affixed together by glue, welding, screws and/or other conventional fastening means, where some of the parts transmit light and some do not, or where different parts transmit light in different ways. In some embodiments, flexible member 74 and/or cover 72 may be double molded and include filters which prevent transmission of light more directly from illuminators 76 to image sensors 78 through cover 72. Optical windows may be provided by cover 72 or elsewise at each image sensor 78.
  • Eye tracking device 10 may emit light from illuminators 76 that illuminate at least part of at least one of the user's eyes. One or more image sensors 78 may then capture an image comprising at least the portion of the eye as illuminated by illuminators 76. This captured image may be transmitted via the flexible member 74 to processing devices (e.g., control unit 22 or other processor, perhaps in a device separate from frame 12), where the direction of the gaze of the user may be determined by analyzing the image data.
  • To determine the gaze direction of the user, a cornea position of one or both eyes of the user may be determined. The cornea position of the user's eye(s) may be analyzed by detecting glints or reflections of the light that is emitted from illuminators 76 onto the eye(s) of the user. In order to obtain a high quality image showing a glint on the user's eye, various combinations of multiple illuminators 76 may be used to emit light. For example, one illuminator 76 may illuminate using infrared light, while another illuminates using another wavelength of light. The image sensors 78 may then capture images when the eye is illuminated with ambient light only, when the eye is illuminated by each illuminator 76 individually, or when the eye is illuminated by two (or more) illuminators 76 at the same time. In such embodiments, image sensors 78 may prepare differential images of the user's eye.
  • Detection of glints may be done with image differentiation techniques (i.e., comparing a first image to a second image to detect a change) and/or standard image analysis algorithms. For example, by toggling an illuminator 76 on and off and capturing an image of the user's eye in each state, the resulting glint may be detected via comparison of the images. However, in some embodiments, glints may be detected by analyzing an one or more images to identify areas of intensity corresponding to a glint. One method of glint detection is outlined in the article “Improved Video-Based Eye-Gaze Detection Method,” by Yoshinobu Ebisawa, published on 4 Aug. 1998, which is hereby incorporated by reference, for all purposes, as if fully set forth herein. Further methods of glint detection are discussed in U.S. Pat. No. 6,577,329 titled “Method and system for relevance feedback through gaze tracking and ticker interfaces” and U.S. Pat. No. 8,292,433 titled “Method and apparatus for communication between humans and devices.” The entire disclosures of the aforementioned patents are hereby incorporated by reference, for all purposes, as if fully set forth herein. Persons skilled in the art will be aware of a number of ways of determining a gaze direction from light reflecting from a user's eye, and the present invention is not limited to the examples recited above.
  • In some embodiments of the invention, a given number of infrared illuminators 76 may be used to illuminate each of a user's eyes. As discussed, other numbers and/or types of illuminators 76 may be used in other embodiments. In such embodiments one or more glints corresponding to each illuminator 76 would be expected. As there may also be other glints resulting from ambient lighting, such as a spotlight or sunlight, etc., different techniques may be used to identify which glints correspond to the illuminators 76, and which ones do not. In some embodiments, an image taken of the user's eye without the illuminators 76 turned on may be compared to one taken with the illuminators 76 turned on to filter out glints caused by ambient light. However, in other embodiments, the size, shape, expected intensity, and expected positions of glints may be used to determine which glints correspond to which illuminators.
  • In other embodiments, a wavelength filter may be used in conjunction with image sensors 78 to filter out wavelengths of light that do not correspond to the wavelengths emitted by illuminators 76. For example, where illuminators 76 emit infrared light, a filter that passes only infrared light through to image sensor 78 may be used. In this way, image sensors 78 may only detect glints resulting from light emitted by the illuminators 76. Conversely, filters may be employed that filter infrared light emitted by illuminators 76 while passing ambient light. Such filters may work well with respect to various artificial sources of ambient light. In some embodiments, lenses 14 may also be configured to block ambient infrared or some other wavelength of light However, in the case of direct sunlight, which includes a spectrum of light comprising both infrared and ultraviolet, filters may not be able to adequately block all of the ambient light and pass only the light emitted by illuminators 76. As a result, glints resulting from ambient light may not be distinguishable from glints resulting from light emitted by illuminators 76 when the illuminator is lit.
  • Ambient light conditions for a user of wearable eye-tracker device 10 may change drastically over time. For example, if the user is facing the sun, his or her eyes may be subject to substantial illumination by sunlight, whereas if the user is facing away from the sun, his/her eyes may be significantly less illuminated. Similarly, if the user is in an indoor environment, the illumination may vary significantly based on the proximity of various light sources. For example if the user is standing directly below a ceiling light, his/her face and eyes may be substantially more illuminated than if he/she were standing adjacent to the ceiling light. Furthermore, in some embodiments, depending on the placement and types of light sources, the levels of ambient light may be different between the two eyes being tracked. For example, the user might be positioned such that his/her left side is in direct sunlight while his/her right side is in a shadow.
  • As discussed, it may be possible to compensate for some changes in ambient light levels by using the built in illuminators 76. It may further be possible to counter the effect of ambient light variations by using light outside the frequency range of light available in the ambient lighting setting. For example in an indoor setting, ultraviolet illuminators 76 may be used to illuminate the user's eyes with ultraviolet light that is not present or is present at lower levels in the indoor lighting. Similarly, infrared illuminators 76 may be used on some embodiments, given that infrared light is typically present at low levels in indoor settings. In some embodiments, illuminators 76 capable of emitting light over a range of wavelengths may be used. In such embodiments, the device 10 may be programmed to dynamically analyze the spectrum of ambient light, and select a wavelength to be emitted by illuminators 76 that is higher or lower than the detected spectrum or otherwise confined to a certain part of the detected spectrum, in which case sensor 78 may also be dynamically adjusted by the device 10. Alternatively, more than one sensor 78 may be provided of different types attuned to different wavelengths.
  • In another embodiment, a wavelength of light may be employed which has a high level of absorption in water, such as 940 nm. This concept is discussed in European Patent Application No. 12192370.0, the entire disclosure of which is hereby incorporated by reference, for all purposes, as if fully set forth herein. At this wavelength the dark pupil effect of the eye may be at its maximum as a majority of light entering the eye will be absorbed. Additionally, ambient light levels in the atmosphere at this wavelength are relatively low. Further, ambient light may be addressed by providing glasses lens 14 in a suitable material or configuration to block light at the wavelength visible to sensor 78. Functionally this improves the signal quality as light is blocked from passing through the glasses lens 14.
  • In some embodiments, differential lighting applications may be used. For example, image sensor(s) 78 may capture images of the user's eye 60 times per second and illuminators 76 may be configured to change state (on/off) 60 times per second out of phase with exposure of sensor(s) 78. In such a scenario, every other frame of image data represents the user's eye as illuminated by illuminators 76, with the alternate frames representing the user's eye as illuminated by ambient light. When analyzing the image data, image processing may be employed to determine the difference between two adjacent frames and thus differentiate glints caused by reflection from illuminators 76 from those caused by reflection from ambient light. The intensity or exposure time of illuminators 76 may be adjusted dynamically in some embodiments, for example based on the level of ambient light around the wearer. For example, a higher level of illumination, or longer exposure time, may be applied when more ambient light is present, or a lesser level of illumination, or shorter exposure time, may be applied if glints from ambient light are advantageously used by the sensors. Furthermore, illuminators 76 may be controlled in groups to allow image sensors 78 and image processing algorithms to detect and compensate for noise from ambient light.
  • Ambient light may be used to detect contrasts in captured image data to determine the position of the user's pupil and/or iris. This information may be used together with information based on glints associated with illuminators 76 to determine the direction in which the user is looking Different settings for pulse length and intensity of the illumination emitted from illuminators 76 and the exposure time of image sensors 78 may be used to compensate for brightness of ambient light, and may in particular improve performance in dark or bright ambient light conditions.
  • For example, when there is a high level of ambient light, illuminators 76 may be disabled and the position of the user's pupil may be triangulated based on glints or other image data detected as a result of the ambient light. Thus, image data may be analyzed to determine the location of the cornea using only ambient light. In some cases, illuminators 76 may be used to assist the contrast detection, with short flashes of illumination coordinated with image capture by image sensors 78 may be sufficient to detect the position of the user's pupils. The use of short intensive illumination pulses with shorter exposure times may help avoid effects such as over-exposure, motion blur and rolling shutter effects. For example, in a shopping research scenario, a mobile eye tracker user may move his/her head around quickly while scanning the store shelves for a desired product. Similarly, when there is too little ambient light in the environment, the illuminators 76 may be used to help produce light to make it possible to pick up the contrasts of the eye. Illuminators 76 may thus be configured to emit lower intensity longer pulses with longer exposure times, and/or constantly emitting low intensity light. In one embodiment of the present invention, all illuminators 76 may be activated concurrently such that multiple reflections on the cornea of emitted light may be captured by sensors 78. Device 10 may then use measured pupil position to determine one or more of gaze direction, orientation data, cornea position, and pupil size.
  • In some embodiments, a calibration of at least one of the plurality of illuminators 76, at least one of the plurality of image sensors 78, or an algorithm of the control unit 22 may be conducted in controlled lighting conditions to determine the location of the pupil and or cornea. In yet other embodiments, the location of the iris may be tracked based on ambient light alone; for example the lines of the iris may be identified to determine the orientation of the eye. Various modes that rely on ambient light only may be activated based on a number of conditions. For example, such a mode may be activated when battery power is low so as to conserve power by disabling illuminators 76. Furthermore, the mode may be activated when ambient lighting reaches a level to where the accuracy of glint-based tracking is at least comparable to the accuracy of another available mode. Persons skilled in the art will appreciate that a number of different configurations may be used to obtain image data, and that the invention is not limited to the examples recited above. Additional calibration methods will be further discussed herein.
  • Accuracy of portable eye tracker device 10 may be maximized by ensuring that the glints are of optimal size. Glints that are too small may be difficult to detect and therefore reduce accuracy. On the other hand, glints that are too large may be difficult to accurately place and may therefore similarly reduce performance by interfering with pupil detection or otherwise. The size of the glints may be affected by the intensity of light emitted by illuminators 76. The size of the glints may further be affected by the sensitivity settings, aperture and/or exposure time for image sensors 78. The setting(s) for image sensors 78 may be adjusted to compensate for ambient light, and the intensity of illuminators 76 may in turn be adjusted to obtain the optimal glint size. Furthermore, the intensity of illuminators 76 and the settings for image sensors 78 may be balanced to obtain the optimal contrast level of the glints such that they can be easily detected. A person skilled in the art will appreciate that the size, contrast and other characteristics of glints may be optimized in a number of ways, and that the invention is not limited to the examples recited above.
  • The light conditions for each eye can be determined by analyzing images captured by image sensors 78 or using external light sensors, or a combination of both. The result of this analysis may be used to control the eye tracker settings for each eye in real-time. In some embodiments of the invention, the relevant light levels may be determined by light sensors embedded in portable eye tracker device 10. In other embodiments, the light levels may be determined using one or more scene cameras 18. By individually controlling the illuminators in real-time based on the current light conditions for each eye, overall performance of the eye tracker device 10 may be improved compared to situation in which the settings are based on only one eye or an average of both eyes.
  • After the glints corresponding to illuminators 76 have been identified in the image data captured by image sensor 78, the location of such glints relative to the user's eye is determined using known image processing techniques. Once the glint locations are determined, they may be analyzed to determine the position of the user's cornea. In some embodiments, the locations of the glints may be mapped on to a three-dimensional model of the human eye. For example, the glints may be mapped onto the cornea. In some embodiments the cornea may be assumed to be a perfect sphere; the locations of the glints may be used to determine the location of the cornea relative to the pupil. This location of the cornea relative to the pupil may in turn be used to determine the direction of the gaze and thereby the optical axis of the eye. Various means of determining a gaze direction based on glints are known in the art and a person skilled in the art would appreciate that the present invention is not limited to the examples recited above.
  • In some embodiments of the invention, more than one image sensor 78 is provided for capturing images of each eye. Where two image sensors 78 are used for each eye, this may be referred to as “stereo mode.” By capturing images of an eye from multiple points of view, additional information may be determined by processor such as control unit 22 or other processor, such as the distance of the eye from each image sensor 78. Further, by operating more than one image sensor 78 there is a level of redundancy in the system whereby it may still function even if one or more image sensor 78 cease to function.
  • Any number of image sensors and/or configuration thereof may be combined with multiple illuminators 76 to operate in various configurations to optimize the reliability of the system. For example, device 10 may be configured to try various configurations of illuminators 76 (e.g., cycle through various illumination patterns/sequences) to determine which configuration of illuminators 76 creates the best image at each image sensor 78. Illuminators 76 may then be configured to change state (on/off) at a frequency such that each image sensor 78 takes a picture in optimal illumination. For example, if there are three image sensors 78 for each eye, and each image sensor 78 captures 30 images per second, illuminators 76 may be configured to change state 90 times per second so that each image sensor 78 can capture an image in a dedicated illumination setting. This may further be used to provide an increased frame rate for the gaze data. Further, using more than one image sensor 78 for an eye reduces the necessity of calibration between the device 10 and the user, i.e., it may be possible to operate the device 10 without user calibration in some instances due to the extra data gathered by having more than one image sensor 78.
  • More than one image sensor 78 may also allow for compensation of physical movement of the device 10. For example, device 10, when configured as a pair of glasses, may slide down the user's nose, and the user may in turn push it back up. Device 10 may also move relative to the user's head after rapid movement of the user's head or for any other reason. This type of movement of device 10 relative to the user's head may reduce or eliminate any accuracy gained from prior calibration. Using multiple image sensors 78 may improve accuracy without the need for recalibration. For example when two image sensors 78 are used for each eye, each glint is detected from two perspectives and it is possible to estimate the location of the cornea with greater accuracy even in cases where device 10 has moved relative to the user's head after any initial calibration.
  • In some embodiments of the present invention, device 10 further includes one or more movement sensor or positioning device 25. Movement sensor or position device 25 may include one or more of a gyroscope; an accelerometer; a compass, a GPS or other satellite receiver; GLONASS compass; or any other location, positioning, or direction sensor. Movement sensor or positioning device 25 may enable tracking of the position and/or orientation of device 10 itself and in turn the location or position of the user's head. This may allow the device 10 to account for head movement and adjust gaze direction data based on the information about head movement to provide a better estimate of the direction of the gaze. A gaze direction determined based on the orientation of the user's eyes may be relative to the orientation of the user's head. Information about the gaze direction based on the user's eyes may therefore be augmented with information about the orientation of the user's head. For brief moments, this augmentation may be performed using an accelerometer or a gyroscope. However, the inaccuracies of these devices may lead to drift if they are used to determine the orientation over time. In some embodiments information about gaze direction based on the user's eyes and/or information about the orientation of the user's head may be further augmented using information from a compass which provides orientation information relative to absolute points of reference. Additionally information about the change in orientation of the device may be helpful when conducting fixation filtering of gaze data. Assuming that objects in the surrounding stay relatively stationary it may be much easier to determine that the user wearing the device fixates at a stationary object while moving their head if device orientation data is available.
  • In other embodiments, movement sensor or positioning device 25 may be embedded in an apparatus connected with device 10 or any portion of device 10. For example, movement sensor or positioning device 25 may be connected with device 10 or any portion of device 10 wirelessly or with a cable. In some embodiments, such movement sensor or positioning device 25 may be carried in a backpack worn by the user, or otherwise carried by the user. When movement sensor or positioning device 25 is embedded in the device 10 it may be able to provide more accurate information about the location of the user's gaze. A number of different systems can provide or account for the required latency and accuracy. In particular, gyroscopes and compasses may provide more accurate information if embedded in the device 10 itself.
  • Furthermore, information from movement sensor or positioning device 25 may be used to stabilize image data from scene camera(s) 18. For example, scene camera 18 may capture image data approximating the view seen by the user. Information from movement sensor or positioning device 25 may be used to stabilize this image data. Information about the user's gaze may further be used to crop or otherwise adjust this image data to more accurately represent the gaze direction of the user. Image stabilization or compensation may include line-shifting of the video data. There are a number of well-known image stabilization methods that involve the use of a gyroscope or accelerometer, and a person skilled in the art will appreciate how these methods may be combined with embodiments of the present invention. Furthermore, time based compensation may be applied to the image data to account for image capture delay inherent in scene camera 18. In particular, images captured by video cameras, rolling shutter cameras as well as CCD type cameras may not show the correct view for when a gaze point was obtained due to image capture delay. Time compensation may differ depending on the type of scene camera 18 and/or where in the image the gaze point is positioned.
  • Image processing and stabilization may in some embodiments be performed in real-time on device 10, or on an external device, as the data is captured. In other embodiments, the image data may merely be stored on device 10 or transferred to an external device (not shown), and the image stabilization or other processing may be performed later based on the captured data. The approach taken may depend on the processing power available in device 10, as well as the energy available from on-board or otherwise connected power source(s). In particular, certain types of processing may require large amounts of computing power that in turn consume a lot of battery capacity. In some embodiments device 10 may be configurable to either optimize battery capacity or to optimize for real-time processing.
  • Furthermore, elements in the captured image or video from scene camera 18 may be analyzed to determine the orientation of a user's head and the speed of movement of the user's head. By analyzing the relative position of elements in successive images or video, adjustments to the calculation of a gaze direction may be made to compensate for movement. In further embodiments, image sensors 78 and/or the scene camera 18 may utilize a rolling shutter to further improve the accuracy of the accuracy of determining the orientation of the users head. By determining the orientation and movement of a user's head in combination with the readout information of rows of image sensors 78 and scene camera 18, the determined gaze direction may be overlaid on an image captured by scene camera 18 in such a way that the gaze direction may be corrected to reflect the actual scene present in the user's field of view at the time the gaze direction was calculated.
  • For example, device 10 may calibrate captured data from scene camera 18 with gaze data derived from the sensors 78 so as to more correctly reflect where and when a user was looking at a particular time in relation to captured data from scene camera 18. In embodiments of the invention, this approach may be adopted where image data captured by scene camera 18 has been distorted due to, for example, rolling shutter distortion or vibrations. Further embodiments include considering instantaneous movement data of the device and calibrating accordingly by utilizing a motion sensor and/or other sensor.
  • Video from scene camera 18 may be analyzed to identify objects in the user's field of view. By identifying the objects and the distance of the objects from the user, more accurate information regarding the direction and target of the gaze may be determined. Computer vision algorithms may be used to detect objects from the image data. In some embodiments, multiple scene cameras 18 may be used to provide stereo vision and more accurate computer vision algorithms. As with image stabilization and other processing, object identification may be done in real-time or as post processing depending on the computing power and power capacity available on device 10. Alternatively or additionally, scene camera 18 may be a depth camera measuring the distance to objects within the field of view of the person wearing the device 10. The depth camera may also determine the intensity level of the objects, thus also provide a grey scale image.
  • Furthermore, image analysis may be used to determine the location of the user based on identification of objects detected within the user's field of view. In other words, an object within the user's field of view, e.g., a landmark or object known to be associated with a given place or type of place may reveal the user's location. For example an object or other indicia, such as signage, types of products, pricing labels products, etc. may indicate that the user is present within a particular retail store or at least a type of retail store. As another example, scene camera 18 or another scanning device connected to device 10 may be configured to scan bar codes that appear within the user's field of view, which may reveal that the user is present within a retail store or other known location or type of location. Such location information may be compounded with information from location sensors and movement sensors to determine a path taken by a user to get to his/her present location and/or to navigate around that location. Where a user is in the vicinity of a television or display, for example displaying a computer game, the game may be able to process eye tracking input provided by device 10; image analysis may be used to determine the direction of the user's gaze relative to the television or display.
  • Furthermore, image analysis of images from scene camera 18 may be utilized for simultaneous localization and mapping (SLAM). SLAM is the process of building a map of a location while simultaneously mapping the position of a device within that map. SLAM is frequently used by robots and similar appliances and may include an image sensor for capturing images of the robot's environment for mapping. A device according to an embodiment of the present invention may also be used with SLAM.
  • Where the motion sensors are used to detect motion, there may be considerable drift within the data when they are relied on for longer periods of time. The information from these motion sensors may therefore be corrected by location information determined based on objects detected from scene camera(s) 18. Objects detected using scene camera(s) 18 may include characteristic structures or objects as well as bar codes. In other embodiments, sound information detected via microphone 20 may be used to determine a location. For example ultrasound emitters may be placed at various points throughout a particular location, such as a retail store, and microphone 20 may be used to determine the closest emitter. Alternatively the ultrasound source may be mounted on device 10 and the microphone used to determine the distance to the closest object in the direction of the ultrasound source. In addition or in the alternative, microphone 20 may be used to detect other ambient sounds and such information may be used, at least in part, to determine a location or type of location in which the user is present. In some embodiments, an RFID tag reader may be included in device 10 so that RFID tags can be used to determine the user's location. As an additional example, Wi-Fi signals and/or other communication signals may be received and triangulated by appropriate transceivers and logic onboard the device 10 to determine the user's location. A person skilled in the art will appreciate that the invention is not limited to the examples recited above and that a number of location identification means may be used to aggregate or determine location information about the wearer.
  • In some embodiments, one or more additional devices may be embedded in or coupled with device 10. For example, scene camera 18 may be used to record images in an area in which the user might move and/or look. A gyroscope may be used for a compass-like feature to identify which direction device 10 is pointing and thus in which direction the user is looking Image sensors 78 in the device may then identify the angle and distance of the user's gaze point based on the direction of where the head is pointing. Information from these multiple sensors in combination may be used to compute a vector representing the wearer's gaze. This vector may be transformed and visualized in the view of scene camera 18. The vector may be used in some embodiments to provide a heat map based on information about where the user's gaze has been focused. A display 30 provided on device 10 may allow the user to view this or other visual data provided by device 10. Merely by way of example, such display 30 could include an LCD screen, an LED screen, a prism projector, and/or other display technologies.
  • In some embodiments of the invention, a gaze-vector can be used to simulate a perspective from the user's point of view along the gaze vector. As described above, the gaze-vector can be used to stabilize an image from scene camera 18 attached to device 10. However, scene camera 18 may not capture video of sufficient quality or have the ability to transmit the video in sufficient quality. The gaze vector may therefore be used along with video data from other cameras, such as stationary or track cameras to prepare a composite view that mimics the view from scene camera 18. Similarly, gaze vectors for one or more users may be displayed in an overview perspective. This application of the present invention may be particularly relevant to sporting events. For example in a football game, a gaze vector for an individual player may be displayed along with gaze vectors for some or all of the other players. Furthermore, a picture-in picture of the scene-camera view or an approximated scene-camera view may be displayed for the player that has the ball. A similar application may be used in large coordinated police actions. A person skilled in the art will appreciate that the invention is not limited to the examples recited above, and that applications with different sports and in other situations may be useful.
  • The gaze vector may further be used to optimize video data from one or more scene cameras 18 in a number of ways. In one embodiment of the invention, the gaze vector may be used to determine focus and light settings to scene camera 18. For example, scene camera 18 may be focused on the target of the gaze vector. Furthermore, the lighting levels of scene camera 18 may be optimized for the target area of the determined gaze direction. In embodiments of the invention where computer vision is used to detect objects captured by scene camera 18, information about the distance of the objects from the user may be determined and used to further improve the exposure or lighting settings of scene camera 18. In some embodiments, a gaze target area may be determined within the image data representing images of at least a portion user's field of view. In some of these embodiments, the gaze target area may comprise less than five percent, less than ten percent, less than 15 percent, less than 20 percent, less than 25 percent, less than 30 percent, less than 35 percent, less than 40 percent, less than 45 percent, or less than 50 percent of the image data in an image from the scene camera. The control unit 22 may then control the scene camera to adjust at least one of focus or light sensitivity based on, and/or within, the gaze target area.
  • However, in other embodiments, adjustment of scene camera 18 may not be necessary. For example when a user is driving a car in bright daylight scene camera 18 may capture the dashboard of the car and the view through the windshield. The objects seen through the windshield may be illuminated at a much higher level than the dashboard, and are naturally much further away. Using standard autofocus and light detection, scene camera 18 may be adjusted based on the average brightness in the detected image and focused to the object in the center of the camera view. However, when the gaze vector is used, the image may be focused on the dashboard when the user looks at the dashboard, and focused on the road when the user looks at the road. Similarly, the lighting levels of the captured video may be adjusted to be appropriate for the dashboard when the user looks at the dashboard, and to be appropriate for the road when the user looks at the road.
  • In some embodiments, the gaze data may be used to prioritize areas of the image for more detail. Based on the video compression used, priority may be given to the area of the image targeted by the gaze. In other embodiments, the video stream may be divided into two or more feeds. One low quality feed may include the entire field of view of the scene camera, whereas a high quality feed may include a small area around the target of the gaze vector. In another embodiment a matrix of video feeds may be used and their bitrate may be dynamically adapted based on the location of the gaze vector. A person skilled in the art will appreciate that the video quality may be adjusted in a number of different ways and that the invention is not limited to the examples recited above.
  • In further embodiments, device 10 may include a speaker for emitting sounds to the user. The speaker may be placed on device 10 in proximity to the user's ear.
  • In embodiments where a processor is used to detect objects in images captured by scene camera 18, the gaze vector can be used to select only a subset of the image data around the gaze target to process and thus reduce the amount of processing work to improve feedback time, battery time etc.
  • Embodiments of the present invention may employ action triggers which cause an action to be performed by device 10, some subcomponent thereof, or a connected system, such as a computer, tablet, television, and/or game machine. According to some embodiments, an action trigger may be enacted by device 10 in many possible ways, including:
      • Images captured by sensor 78 and processed by device 10 or a connected system, resulting in a detection of a present eye, an absent eye, a blink, a fixation, a saccade, a direction of the user's gaze or the movement of a user's gaze in a predetermined pattern.
      • Images captured by scene camera 18 and processed by device 10 or a connected system, resulting in a detection of changes in light, identified object(s), identified pattern(s) and identified gesture(s) using computer vision algorithms.
      • A switch or button based trigger. For example, a physical button on device 10 or another device.
      • An audio based trigger. For example, sounds, words or commands spoken by a user and detected by microphone 20.
      • An accelerometer- or gyroscope-detected action trigger, such as a nod or other head movement.
      • A combination of actions described above.
  • Embodiments of the present invention include calibration techniques whereby external objects may be used to facilitate the calibration process of the device 10. Merely by way of example, illuminators 76, image sensors 78, and/or algorithms of control unit 22 may be adjusted during such calibration. In some embodiments, an external device such as a printed medium, television, or other display may contain a feature readily identifiable by device 10. In some embodiments, the feature may be a specific frame, QR code, invisible feature (i.e., infrared feature), and/or other visible feature. The feature may contain an identification code which, when recognized by device 10, allows the device 10 to connect to the external device by way of a communication protocol such as the internet, Bluetooth, wi-fi or any other communications protocol. The external device may then enter a calibration mode whereby icons for calibration are displayed on the screen and calibration instructions shown or emitted through speakers. Specific calibration processes are well known but typically include an ordered display of icons on a screen, which a user then gazes at and device 10 determines the gaze direction relative to each displayed icon.
  • By way of specific example, some embodiments may perform a calibration method whereby when device 10 is worn by a user, and a pattern is placed in front of the user such that the pattern is within the field of view of the scene camera 18. Scene camera 18 records images of the pattern, while image sensors 78 capture images of the user's eye(s). The processing device analyses the pattern and determines known components of the pattern. The processing device analyses the gaze direction of the user utilizing the images captured by the image sensors 78 and matches the gaze direction against the known components of the pattern. As the components are known, the processing device may determine the offset between the location of the components and the determined gaze direction of the user. By knowing this offset, the device is calibrated for the particular user and can consider the offset when determining the user's gaze direction relative to other objects.
  • Further, device 10 may utilize in an algorithm or mathematical model information obtained from analyzing the relationship between a user's gaze direction and known components of the pattern or similar. By way of example device 10 may set a value of a parameter used in an algorithm or eye model used to determine a gaze direction, as would be readily understood by a person skilled in the art.
  • In some embodiments, additional functionality may be provided when device 10 is used in the context of a retail or consumer purchasing environment, such as a supermarket. The device may preload information such as personal calibration data to match the profile of a user. Once device 10 is worn by a user it may change from a low power mode to normal power mode and enter operational mode. Device 10 may allow a user to interactively manage their shopping bill by adding and removing items from a virtual shopping cart. For example, when a user places an item into their shopping cart, they can look at a bar code on the item and swipe right to left with their finger across the barcode to indicate that the item should be added to the list. Device 10 may recognize the gesture and add the item to a virtual list either on device 10 or at a remote location in communication with device 10. This virtual list may be controlled by the user for example by removing items by swiping left to right across a barcode, swiping top to bottom to obtain further information about an item and gazing at a shopping cart and touching a predetermined location to hear feedback about the list or other information. Upon checking out from the store, the virtual list may be retrieved by a cashier or automatic machine and the user may pay directly for the items in the shopping cart. In a further embodiment, the user may exit the store directly and upon exiting the store, the value of the list may be deducted from a monetary source such as a credit card or account.
  • In another embodiment, device 10 may perform optical character recognition (OCR) on image data including text that is read by a user. For example, scene camera 18 may record images of text read by a user as determined by the user's gaze direction. Device 10, or a computing device in communication with device 10, may perform OCR on the images of the text to determine the words comprising the text. OCR techniques are well understood and known in the art. Once text has been analyzed using an OCR technique, it may be sent to a text-to-speech functionality that reads out the text loud, translated to a result presented to the user in real time, and/or saved as text data or a string that may be readily manipulated and understood by a computing device. The text may be saved for retrieval by a user, or utilized by a computing device or service to understand habits of the user. For example, the text may indicate a product or service of particular desire to the user. In embodiments, the text data may be utilized to customize advertisements or the like to be displayed to the user.
  • In embodiments of the invention, device 10 may consider known information such as the size of a barcode or standard object. For example a product in a store may include a number of barcodes, one larger barcode containing the product identifier or SKU, and one smaller barcode containing the serial number of the particular item in that box. Device 10 may be configured to direct image sensors 78 to only read the larger code when preparing a shopping list. This may be based on the gaze focus distance for better accuracy. Further, many software solutions such as object recognition libraries require video of a standard size. Accordingly, device 10 may capture images or video only of the required size, or crop captured images or video such that they are in the required size.
  • FIG. 5 shows a block diagram of one possible method 500 of the invention for determining a gaze direction of a user using a wearable frame as described herein. At block 510, as discussed above, device 10 may be calibrated prior to initial use. At block 520, non-image information such as that from movement sensor or positioning device 25 may be acquired by control unit 22. Other types of non-image information may also be received as described herein (e.g., information from microphone 20). At block 530, image information from scene camera 18 may be received by control unit 22.
  • At block 540, control unit 22 may activate illuminators 76 according to settings established during calibration at step 510. At block 550, image data may be received by control unit 22 from image sensors 78. At block 560, control unit 22 may determine if from the information it has received if a gaze direction can be determined as described above. If not, at block 570, illuminators 76, image sensors 78, and/or other components of device 10 may be adjusted as described herein, and then method 500 returns to block 520. In some embodiments, method 500 may return to another step such as block 530 or block 540 if the gaze direction cannot be determined. If control unit 22 can determine gaze direction from the information it has received, at block 580 the gaze direction is determined. Method 500 may repeat at regular or irregular intervals to re-determine gaze direction as needed.
  • FIG. 6 is a block diagram illustrating an exemplary computer system 600 in which embodiments of the present invention may be implemented. This example illustrates a computer system 600 such as may be used, in whole, in part, or with various modifications, to provide and/or control the functions of control unit 22, illuminators 76, image sensors 78 and/or other components of the invention such as those discussed above. For example, various functions of control unit 22 may be controlled by the computer system 600, including, merely by way of example, controlling illuminators 76, receiving images from image sensors 78, processing data from image sensors 78, etc.
  • The computer system 600 is shown comprising hardware elements that may be electrically coupled via a bus 690. The hardware elements may include one or more central processing units 610, one or more input devices 620 (e.g., a mouse, a keyboard, etc.), and one or more output devices 630 (e.g., a display device, a printer, etc.). The computer system 600 may also include one or more storage device 640. By way of example, storage device(s) 640 may be disk drives, optical storage devices, solid-state storage device such as a random access memory (“RAM”) and/or a read-only memory (“ROM”), which can be programmable, flash-updateable and/or the like.
  • The computer system 600 may additionally include a computer-readable storage media reader 650, a communications system 660 (e.g., a modem, a network card (wireless or wired), an infra-red communication device, Bluetooth™ device, cellular communication device, etc.), and working memory 680, which may include RAM and ROM devices as described above. In some embodiments, the computer system 600 may also include a processing acceleration unit 670, which can include a digital signal processor, a special-purpose processor and/or the like.
  • The computer-readable storage media reader 650 can further be connected to a computer-readable storage medium, together (and, optionally, in combination with storage device(s) 640) comprehensively representing remote, local, fixed, and/or removable storage devices plus storage media for temporarily and/or more permanently containing computer-readable information. The communications system 660 may permit data to be exchanged with a network, system, computer and/or other component described above.
  • The computer system 600 may also include software elements, shown as being currently located within a working memory 680, including an operating system 684 and/or other code 688. It should be appreciated that alternate embodiments of a computer system 600 may have numerous variations from that described above. For example, customized hardware might also be used and/or particular elements might be implemented in hardware, software (including portable software, such as applets), or both. Furthermore, connection to other computing devices such as network input/output and data acquisition devices may also occur.
  • Software of computer system 600 may include code 688 for implementing any or all of the function of the various elements of the architecture as described herein. For example, software, stored on and/or executed by a computer system such as system 600, can provide the functions of control unit 22, illuminators 76, image sensors 78, and/or other components of the invention such as those discussed above. Methods implementable by software on some of these components have been discussed above.
  • The invention has now been described in detail for the purposes of clarity and understanding. However, it will be appreciated that certain changes and modifications may be practiced within the scope of the appended claims.

Claims (20)

    What is claimed is:
  1. 1. A portable eye tracker device comprising:
    a frame adapted for wearing by a user;
    at least one optics holding member, wherein the optics holding member comprises:
    at least one illuminator configured to selectively illuminate at least a portion of at least one eye of the user; and
    at least one image sensor configured to capture image data representing images of at least a portion of at least one eye of the user; and
    a control unit configured to:
    control the at least one illuminator for the selective illumination of at least a portion of at least one eye of the user;
    receive the image data from the at least one image sensor; and
    calibrate at least one illuminator, at least one image sensor, or an algorithm of the control unit.
  2. 2. The portable eye tracker device of claim 1, wherein the control unit is further configured to:
    determine a gaze target area for the user based at least in part on the image data.
  3. 3. The portable eye tracker device of claim 1, further comprising:
    a scene camera facing away from the user and configured to capture image data representing images of at least a portion of the user's field of view.
  4. 4. The portable eye tracker device of claim 3, wherein the control device is further configured to:
    determine a gaze target area within the image data representing images of at least a portion user's field of view wherein the gaze target area comprises less than ten percent of the image data in an image from the scene camera; and
    control the scene camera to adjust at least one of focus or light sensitivity based on the gaze target area.
  5. 5. The portable eye tracker device of claim 3, wherein the control unit is further configured to:
    receive image data from the scene camera;
    determine, automatically, a specific pattern or item is within the image data received from the scene camera; and
    calibrate at least one illuminator, at least one image sensor, or an algorithm of the control unit based at least in part on the image data having the specific pattern or item.
  6. 6. The portable eye tracker device of claim 5, wherein the specific pattern is displayed on a printed medium or provided by a display device.
  7. 7. The portable eye tracker device of claim 1, wherein calibrating at least one of the plurality of illuminators or at least one of the plurality of image sensors comprises a selection from a group consisting of:
    adjusting a brightness of at least one illuminator;
    determining an offset between a calculated gaze direction and an actual gaze direction;
    determining a value of a parameter used by an algorithm determining the user's gaze target area;
    adjusting a wavelength of light produced by at least one illuminator;
    adjusting a sensitivity of at least one image sensor;
    adjusting a focus of at least one image sensor; and
    changing at least one parameter of the algorithm of the control unit.
  8. 8. The portable eye tracker device of claim 1, further comprising:
    a movement sensor configured to determine movement of the frame.
  9. 9. The portable eye tracker device of claim 1, wherein the control unit is further configured to:
    calibrate at least one illuminator, at least one image sensor, or set a value of a parameter used by an algorithm determining the user's gaze target area.
  10. 10. The portable eye tracker device of claim 8, wherein the control unit is further configured to:
    determine a gaze target area for the user based at least in part on the image data and information received from the movement sensor.
  11. 11. The portable eye tracker device of claim 1, wherein the control unit is further configured to:
    control at least one of illuminator to selectively illuminate at least a portion of at least one eye of the user using a light of a first wavelength;
    receive a first set of image data from at least one image sensor, wherein the first set of image data represents images of at least a portion of at least one eye of the user captured when the user's eye is exposed to the light of the first wavelength;
    determine that a gaze target area cannot be determined based on the first set of image data;
    receive information about ambient light illuminating at least a portion of at least one of the user;
    determine a second wavelength based on the information about the ambient light, wherein the second wavelength is the same or different than the first wavelength;
    control at least one of illuminator to selectively illuminate at least a portion of at least one eye of the user using a light of a second wavelength;
    receive a second set of image data from at least one image sensor, wherein the second set of image data represents images of at least a portion of at least one eye of the user captured when the user's eye is exposed to the light of the second wavelength; and
    determine the gaze target area for the user based at least in part on the second set of image data.
  12. 12. The portable eye tracker device of claim 1, wherein the control unit is further configured to:
    control a first subset of the at least one illuminator to selectively illuminate at least a portion of at least one eye of the user;
    receive a first set of image data from at least one image sensor of the first subset, wherein the first set of image data represents images of at least a portion of at least one eye of the user captured when at least one eye of the user is exposed to light from the first subset of the at least one illuminator;
    determine that a gaze target area cannot be determined based on the first set of image data;
    control a second subset of the at least one illuminator to selectively illuminate at least a portion of at least one eye of the user;
    receive a second set of image data from at least one image sensor of the second subset, wherein the second set of image data represents images of at least a portion of at least one eye of the user captured when at least one eye of the user is exposed to light from the second subset of the at least one illuminator; and
    determine the gaze target area for the user based at least in part on the second set of image data.
  13. 13. The portable eye tracker device of claim 1, wherein:
    the portable eye tracker device further comprises a positioning device configured to determine a position of the portable eye tracker device; and
    the control unit is further configured to determine a gaze target area for the user based at least in part on the image data and information received from the positioning device.
  14. 14. The portable eye tracker device of claim 1, wherein the control unit is further configured to:
    receive image data from at least one image sensor;
    determine a level of ambient light from the image data;
    adjust a brightness or exposure time of at least one illuminator based at least in part on the level of ambient light.
  15. 15. The portable eye tracker device of claim 1, further comprising:
    a display viewable by the user.
  16. 16. The portable eye tracker device of claim 1, wherein the control unit is further configured to:
    receive image data from at least one image sensor representing images including areas besides the user's eyes.
  17. 17. A method of determining a gaze direction for a user, comprising:
    activating at least one illuminator on a frame worn by a user to selectively illuminate at least a portion of at least one of the user's eyes;
    receiving image data representing images at least a portion of at least one of the user's eyes from at least one image sensor on the frame;
    calibrating at least one illuminators, at least one image sensor, or an algorithm of an associated control unit;
    determining a gaze target area for the user based at least in part on the image data.
  18. 18. The method of claim 17, further comprising:
    receiving image data from a scene camera configured to capture image data representing images of at least a portion of a view of the user;
    determining, automatically, a specific pattern or item is within the image data received from the scene camera; and
    calibrating at least one illuminator, at least one image sensor, or an algorithm of the control unit based at least in part on the image data having the specific pattern or item.
  19. 19. A non-transitory machine readable medium having instructions thereon for determining a gaze direction for a user, the instructions executable by a processor for at least:
    activating at least one illuminator on a frame worn by a user to selectively illuminate at least a portion of at least one of the user's eyes;
    receiving image data representing images at least a portion of at least one of the user's eyes from at least one image sensor on the frame;
    calibrating at least one illuminators, at least one image sensor, or an algorithm of an associated control unit; and
    determining a gaze target area for the user based at least in part on the image data.
  20. 20. The non-transitory machine readable medium of claim 19, wherein the instructions are further executable for at least:
    receiving image data from a scene camera configured to capture image data representing images of at least a portion of a view of the user;
    determining, automatically, a specific pattern or item is within the image data received from the scene camera; and
    calibrating at least one illuminator, at least one image sensor, or an algorithm of the control unit based at least in part on the image data having the specific pattern or item.
US14281676 2013-09-03 2014-05-19 Portable eye tracking device Active 2034-12-25 US9710058B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US201361873154 true 2013-09-03 2013-09-03
US14281676 US9710058B2 (en) 2013-09-03 2014-05-19 Portable eye tracking device

Applications Claiming Priority (9)

Application Number Priority Date Filing Date Title
PCT/US2014/038651 WO2015034560A1 (en) 2013-09-03 2014-05-19 Portable eye tracking device
US14281676 US9710058B2 (en) 2013-09-03 2014-05-19 Portable eye tracking device
KR20167008758A KR101882594B1 (en) 2013-09-03 2014-05-19 Portable eye tracking device
KR20187020804A KR20180086511A (en) 2013-09-03 2014-05-19 Portable eye tracking device
CN 201480060817 CN105960193A (en) 2013-09-03 2014-05-19 Portable eye tracking device
PCT/US2014/038653 WO2015034561A1 (en) 2013-09-03 2014-05-19 Portable eye tracking device
KR20167008761A KR20160111019A (en) 2013-09-03 2014-05-19 Portable eye tracking device
US15277225 US20170017299A1 (en) 2013-09-03 2016-09-27 Portable eye tracking device
US15650445 US20180131850A1 (en) 2013-09-03 2017-07-14 Portable eye tracking device

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15650445 Continuation US20180131850A1 (en) 2013-09-03 2017-07-14 Portable eye tracking device

Publications (2)

Publication Number Publication Date
US20150062323A1 true true US20150062323A1 (en) 2015-03-05
US9710058B2 US9710058B2 (en) 2017-07-18

Family

ID=52582484

Family Applications (9)

Application Number Title Priority Date Filing Date
US14281587 Active US9041787B2 (en) 2013-09-03 2014-05-19 Portable eye tracking device
US14281616 Active 2034-09-17 US9665172B2 (en) 2013-09-03 2014-05-19 Portable eye tracking device
US14281676 Active 2034-12-25 US9710058B2 (en) 2013-09-03 2014-05-19 Portable eye tracking device
US14281599 Pending US20150061995A1 (en) 2013-09-03 2014-05-19 Portable eye tracking device
US14452178 Active US9596391B2 (en) 2013-09-03 2014-08-05 Gaze based directional microphone
US15211183 Pending US20160328016A1 (en) 2013-09-03 2016-07-15 Portable eye tracking device
US15453661 Active US10116846B2 (en) 2013-09-03 2017-03-08 Gaze based directional microphone
US15608058 Pending US20180020137A1 (en) 2013-09-03 2017-05-30 Portable eye tracking device
US15650445 Pending US20180131850A1 (en) 2013-09-03 2017-07-14 Portable eye tracking device

Family Applications Before (2)

Application Number Title Priority Date Filing Date
US14281587 Active US9041787B2 (en) 2013-09-03 2014-05-19 Portable eye tracking device
US14281616 Active 2034-09-17 US9665172B2 (en) 2013-09-03 2014-05-19 Portable eye tracking device

Family Applications After (6)

Application Number Title Priority Date Filing Date
US14281599 Pending US20150061995A1 (en) 2013-09-03 2014-05-19 Portable eye tracking device
US14452178 Active US9596391B2 (en) 2013-09-03 2014-08-05 Gaze based directional microphone
US15211183 Pending US20160328016A1 (en) 2013-09-03 2016-07-15 Portable eye tracking device
US15453661 Active US10116846B2 (en) 2013-09-03 2017-03-08 Gaze based directional microphone
US15608058 Pending US20180020137A1 (en) 2013-09-03 2017-05-30 Portable eye tracking device
US15650445 Pending US20180131850A1 (en) 2013-09-03 2017-07-14 Portable eye tracking device

Country Status (5)

Country Link
US (9) US9041787B2 (en)
EP (2) EP3041400A1 (en)
KR (3) KR20160111019A (en)
CN (3) CN108209857A (en)
WO (2) WO2015034560A1 (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150301599A1 (en) * 2014-04-18 2015-10-22 Magic Leap, Inc. Eye tracking systems and method for augmented or virtual reality
US20150319827A1 (en) * 2014-05-02 2015-11-05 Lg Electronics Inc. Lighting system and control method thereof
US20150319826A1 (en) * 2014-05-02 2015-11-05 Lg Electronics Inc. Lighting system and control method thereof
CN105676456A (en) * 2016-04-06 2016-06-15 众景视界(北京)科技有限公司 Modularized head-mounted electronic device
US20160231573A1 (en) * 2015-02-10 2016-08-11 Daqri, Llc Dynamic lighting for head mounted device
US9596391B2 (en) 2013-09-03 2017-03-14 Tobii Ab Gaze based directional microphone
WO2017053871A3 (en) * 2015-09-24 2017-05-04 Supereye, Inc. Methods and devices for providing enhanced visual acuity
US9727136B2 (en) * 2014-05-19 2017-08-08 Microsoft Technology Licensing, Llc Gaze detection calibration
US9952883B2 (en) 2014-08-05 2018-04-24 Tobii Ab Dynamic determination of hardware

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018064141A1 (en) 2016-09-27 2018-04-05 Tobii Ab Portable eye tracking device
US10082664B2 (en) * 2013-09-23 2018-09-25 Samsung Electronics Co., Ltd. Tracking optics for a mobile device
WO2015104644A3 (en) * 2014-01-10 2015-11-12 The Eye Tribe Aps Light modulation in eye tracking devices
KR20160047196A (en) * 2014-10-22 2016-05-02 삼성전자주식회사 APPARATUS FOR TRACKING EYE POINT OPERABLE AT HIGH intensity of illumination AND LOW intensity of illumination AND METHOD THEREOF
US20160139411A1 (en) * 2014-11-18 2016-05-19 Samsung Electronics Co., Ltd. Wearable device and method for outputting virtual image
EP3112922A1 (en) * 2015-06-30 2017-01-04 Thomson Licensing A gaze tracking device and a head mounted device embedding said gaze tracking device
US9829976B2 (en) 2015-08-07 2017-11-28 Tobii Ab Gaze direction mapping
WO2017053974A1 (en) 2015-09-24 2017-03-30 Tobii Ab Eye-tracking enabled wearable devices
US9924866B2 (en) 2016-01-11 2018-03-27 Heptagon Micro Optics Pte. Ltd. Compact remote eye tracking system including depth sensing capacity
NL2016085B1 (en) 2016-01-13 2017-07-24 Stichting Vu-Vumc Method, system and computer readable medium to determine a strabismus angle between the eyes of an individual.
US9905244B2 (en) * 2016-02-02 2018-02-27 Ebay Inc. Personalized, real-time audio processing
US10088140B2 (en) 2016-06-07 2018-10-02 Heptagon Micro Optics Pte. Ltd. Eye-safe optoelectronic module
FR3053235A1 (en) 2016-06-29 2018-01-05 Inst Mines Telecom Ocular measurement device anywhere a system optically aligned with the line of sight of the user
US20180033201A1 (en) * 2016-07-27 2018-02-01 Google Inc. Low-power mode feature identification at a head mounted display
US20180032103A1 (en) 2016-07-27 2018-02-01 Tobii Ab Wearable device having a display, lens, illuminator, and image sensor
WO2018035160A1 (en) * 2016-08-15 2018-02-22 The Regents Of The University Of California Bio-sensing and eye-tracking system
US10044925B2 (en) 2016-08-18 2018-08-07 Microsoft Technology Licensing, Llc Techniques for setting focus in mixed reality applications
DE102016221123A1 (en) * 2016-10-26 2018-04-26 Bayerische Motoren Werke Aktiengesellschaft Method and apparatus for operating a display system with a data goggles
CN106405843A (en) * 2016-11-24 2017-02-15 天津临境科技有限公司 Projection VR glasses
WO2018100589A1 (en) * 2016-11-30 2018-06-07 Secretary, Department Of Biotechnology Ocular screening
US10121337B2 (en) * 2016-12-30 2018-11-06 Axis Ab Gaze controlled bit rate
US20180267604A1 (en) * 2017-03-20 2018-09-20 Neil Bhattacharya Computer pointer device

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6320610B1 (en) * 1998-12-31 2001-11-20 Sensar, Inc. Compact imaging device incorporating rotatably mounted cameras
US20050175218A1 (en) * 2003-11-14 2005-08-11 Roel Vertegaal Method and apparatus for calibration-free eye tracking using multiple glints or surface reflections
US20100110368A1 (en) * 2008-11-02 2010-05-06 David Chaum System and apparatus for eyeglass appliance platform
US20110069277A1 (en) * 2009-04-01 2011-03-24 Tobii Technology Ab Visual display with illuminators for gaze tracking
US20110211056A1 (en) * 2010-03-01 2011-09-01 Eye-Com Corporation Systems and methods for spatially controlled scene illumination
US20130021373A1 (en) * 2011-07-22 2013-01-24 Vaught Benjamin I Automatic Text Scrolling On A Head-Mounted Display
US20130050070A1 (en) * 2011-08-29 2013-02-28 John R. Lewis Gaze detection in a see-through, near-eye, mixed reality display
US20130114043A1 (en) * 2011-11-04 2013-05-09 Alexandru O. Balan See-through display brightness control
US20130114850A1 (en) * 2011-11-07 2013-05-09 Eye-Com Corporation Systems and methods for high-resolution gaze tracking
US20130300648A1 (en) * 2012-05-11 2013-11-14 Qualcomm Incorporated Audio user interaction recognition and application interface
US20130314303A1 (en) * 2010-02-28 2013-11-28 Osterhout Group, Inc. Ar glasses with user action control of and between internal and external applications with feedback
US20150061996A1 (en) * 2013-09-03 2015-03-05 Tobii Technology Ab Portable eye tracking device
US9474131B2 (en) * 2014-05-02 2016-10-18 Lg Electronics Inc. Lighting device, lighting system and wearable device having image processor

Family Cites Families (85)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3676641A (en) 1971-01-15 1972-07-11 Wallace A Olson Apparatus for assisting in the curing of concrete and for heating
JPH02164335A (en) * 1988-12-16 1990-06-25 Konan Camera Kenkyusho:Kk Eyeball motion analyzing device
US8730129B2 (en) 1990-12-07 2014-05-20 Dennis J Solomon Advanced immersive visual display system
JPH0749744A (en) 1993-08-04 1995-02-21 Pioneer Electron Corp Head mounting type display input device
DE69607376D1 (en) 1995-12-15 2000-04-27 Monsanto Co Method for improved rheological control in cement systems
US6204974B1 (en) 1996-10-08 2001-03-20 The Microoptical Corporation Compact image display system for eyeglasses or other head-borne frames
US6603491B2 (en) 2000-05-26 2003-08-05 Jerome H. Lemelson System and methods for controlling automatic scrolling of information on a display or screen
US6577329B1 (en) 1999-02-25 2003-06-10 International Business Machines Corporation Method and system for relevance feedback through gaze tracking and ticker interfaces
US6592223B1 (en) * 1999-10-07 2003-07-15 Panaseca, Inc. System and method for optimal viewing of computer monitors to minimize eyestrain
US6353422B1 (en) 2000-03-31 2002-03-05 Stephen G. Perlman Virtual display system and method
JP4469476B2 (en) * 2000-08-09 2010-05-26 パナソニック株式会社 Eye position detecting method and the eye position detecting device
US7306337B2 (en) 2003-03-06 2007-12-11 Rensselaer Polytechnic Institute Calibration-free gaze tracking under natural head movement
US8292433B2 (en) 2003-03-21 2012-10-23 Queen's University At Kingston Method and apparatus for communication between humans and devices
US7380938B2 (en) 2003-03-25 2008-06-03 Sarnoff Corporation Apparatus to detect and measure saccade and pupilary changes
US7806525B2 (en) 2003-10-09 2010-10-05 Ipventure, Inc. Eyeglasses having a camera
US7922321B2 (en) 2003-10-09 2011-04-12 Ipventure, Inc. Eyewear supporting after-market electrical components
US8427538B2 (en) 2004-04-30 2013-04-23 Oncam Grandeye Multiple view and multiple object processing in wide-angle video camera
US7561143B1 (en) 2004-03-19 2009-07-14 The University of the Arts Using gaze actions to interact with a display
US7500752B2 (en) 2004-04-28 2009-03-10 Natus Medical Incorporated Diagnosing and training the gaze stabilization system
US20060044461A1 (en) * 2004-08-30 2006-03-02 Popescu-Stanesti Vlad M Digital camera with photoflash controller
US7549743B2 (en) 2005-01-06 2009-06-23 University Of Rochester Systems and methods for improving visual discrimination
EP1933694A4 (en) 2005-10-10 2010-12-22 Tobii Technology Ab Eye tracker having an extended span of operating distances
US9606621B2 (en) * 2006-07-28 2017-03-28 Philips Lighting Holding B.V. Gaze interaction for information display of gazed items
US20090044482A1 (en) 2007-01-30 2009-02-19 Tooman Norman L Wind turbine installation comprising an apparatus for protection of anchor bolts and method of installation
US7584791B2 (en) 2007-02-08 2009-09-08 Halliburton Energy Services, Inc. Methods for reducing the viscosity of treatment fluids comprising diutan
US8519091B2 (en) 2007-04-10 2013-08-27 Air Products And Chemicals, Inc. Polyalkyleneamine adducts as curing agents for thick layer water-based epoxy systems
EP2165234A1 (en) * 2007-06-07 2010-03-24 Panagiotis Pavlopoulos An eyewear comprising at least one display device
US7556377B2 (en) * 2007-09-28 2009-07-07 International Business Machines Corporation System and method of detecting eye fixations using adaptive thresholds
US20090122161A1 (en) * 2007-11-08 2009-05-14 Technical Vision Inc. Image to sound conversion device
CN101583898B (en) * 2007-11-20 2011-12-07 松下电器产业株式会社 Beam scanning-type display apparatus, display method, and automobiles
US8922672B2 (en) * 2008-01-03 2014-12-30 Apple Inc. Illumination systems and methods for imagers
US8786675B2 (en) * 2008-01-23 2014-07-22 Michael F. Deering Systems using eye mounted displays
JP2011519293A (en) 2008-04-14 2011-07-07 ザ・ジョンズ・ホプキンス・ユニバーシティー Vestibular - inspection system and method of oculomotor function
EP2164295A1 (en) 2008-09-16 2010-03-17 Deleage SA Electric heat cable device
US20100074460A1 (en) * 2008-09-25 2010-03-25 Lucent Technologies Inc. Self-steering directional hearing aid and method of operation thereof
US8589034B2 (en) * 2008-10-09 2013-11-19 Angela Karen Kwok System and methods for an automated sun glare block area and sunshield in a vehicular windshield
DK2389095T3 (en) 2009-01-26 2014-11-17 Tobii Technology Ab Detecting the gaze point helped by the optical reference signals
EP2398377A1 (en) 2009-01-30 2011-12-28 Universita' Degli Studi Di Pavia Head impulse testing device for assessing the functionality of the semicircular canals and for the rehabilitation of vestibular loss
US9782068B2 (en) 2009-02-02 2017-10-10 The Johns Hopkins University System for diagnosis and therapy of gaze stability
CN107071688A (en) 2009-06-23 2017-08-18 诺基亚技术有限公司 Method and device for processing audio signals
US8514097B2 (en) * 2009-08-14 2013-08-20 Michael William Boise Glasses with sound activated lights
US20110096941A1 (en) * 2009-10-28 2011-04-28 Alcatel-Lucent Usa, Incorporated Self-steering directional loudspeakers and a method of operation thereof
GB0920809D0 (en) * 2009-11-27 2010-01-13 Qinetiq Ltd Eye tracking apparatus
JP4679661B1 (en) * 2009-12-15 2011-04-27 株式会社東芝 Information presentation apparatus, information presentation method and program
US9229227B2 (en) 2010-02-28 2016-01-05 Microsoft Technology Licensing, Llc See-through near-eye display glasses with a light transmissive wedge shaped illumination system
US9128281B2 (en) * 2010-09-14 2015-09-08 Microsoft Technology Licensing, Llc Eyepiece with uniformly illuminated reflective display
US20120194419A1 (en) * 2010-02-28 2012-08-02 Osterhout Group, Inc. Ar glasses with event and user action control of external applications
US8488246B2 (en) 2010-02-28 2013-07-16 Osterhout Group, Inc. See-through near-eye display glasses including a curved polarizing film in the image source, a partially reflective, partially transmitting optical element and an optically flat film
US8964298B2 (en) * 2010-02-28 2015-02-24 Microsoft Corporation Video display modification based on sensor input for a see-through near-to-eye display
US9129295B2 (en) 2010-02-28 2015-09-08 Microsoft Technology Licensing, Llc See-through near-eye display glasses with a fast response photochromic film system for quick transition from dark to clear
US8472120B2 (en) 2010-02-28 2013-06-25 Osterhout Group, Inc. See-through near-eye display glasses with a small scale image source
KR20130122516A (en) * 2010-04-26 2013-11-07 캠브리지 메카트로닉스 리미티드 Loudspeakers with position tracking
US9760123B2 (en) * 2010-08-06 2017-09-12 Dynavox Systems Llc Speech generation device with a projected display and optical inputs
EP2499960B1 (en) * 2011-03-18 2015-04-22 SensoMotoric Instruments Gesellschaft für innovative Sensorik mbH Method for determining at least one parameter of two eyes by setting data rates and optical measuring device
US8687840B2 (en) 2011-05-10 2014-04-01 Qualcomm Incorporated Smart backlights to minimize display power consumption based on desktop configurations and user eye gaze
US20130028443A1 (en) * 2011-07-28 2013-01-31 Apple Inc. Devices with enhanced audio
US9285592B2 (en) 2011-08-18 2016-03-15 Google Inc. Wearable device with input and output structures
US9213163B2 (en) * 2011-08-30 2015-12-15 Microsoft Technology Licensing, Llc Aligning inter-pupillary distance in a near-eye display system
KR20140059213A (en) * 2011-08-30 2014-05-15 마이크로소프트 코포레이션 Head mounted display with iris scan profiling
US20130083003A1 (en) * 2011-09-30 2013-04-04 Kathryn Stone Perez Personal audio/visual system
US8847988B2 (en) * 2011-09-30 2014-09-30 Microsoft Corporation Exercising applications for personal audio/visual system
US8235529B1 (en) * 2011-11-30 2012-08-07 Google Inc. Unlocking a screen using eye tracking information
US8917453B2 (en) * 2011-12-23 2014-12-23 Microsoft Corporation Reflective array waveguide
US8982014B2 (en) 2012-02-06 2015-03-17 Battelle Memorial Institute Image generation systems and image generation methods
WO2013117727A1 (en) * 2012-02-09 2013-08-15 Universität Zürich System for examining eye movements, particularly the vestibulo-ocular reflex and dynamic visual acuity
US9128522B2 (en) 2012-04-02 2015-09-08 Google Inc. Wink gesture input for a head-mountable device
US9304584B2 (en) 2012-05-31 2016-04-05 Ca, Inc. System, apparatus, and method for identifying related content based on eye movements
US20130326364A1 (en) * 2012-05-31 2013-12-05 Stephen G. Latta Position relative hologram interactions
US9046738B2 (en) * 2012-06-28 2015-06-02 International Business Machines Corporation Digital image capture under conditions of varying light intensity
US9317971B2 (en) * 2012-06-29 2016-04-19 Microsoft Technology Licensing, Llc Mechanism to give holographic objects saliency in multiple spaces
US9854159B2 (en) * 2012-07-20 2017-12-26 Pixart Imaging Inc. Image system with eye protection
EP2731049A1 (en) 2012-11-13 2014-05-14 Tobii Technology AB Eye-tracker
US20140154651A1 (en) 2012-12-04 2014-06-05 Sync-Think, Inc. Quantifying peak cognitive performance using graduated difficulty
US9977492B2 (en) 2012-12-06 2018-05-22 Microsoft Technology Licensing, Llc Mixed reality presentation
KR20140090552A (en) 2013-01-09 2014-07-17 엘지전자 주식회사 Head Mounted Display and controlling method for eye-gaze calibration
US20140191927A1 (en) * 2013-01-09 2014-07-10 Lg Electronics Inc. Head mount display device providing eye gaze calibration and control method thereof
CN105073073A (en) * 2013-01-25 2015-11-18 胡海 Devices and methods for the visualization and localization of sound
DE102014109079A1 (en) * 2013-06-28 2014-12-31 Harman International Industries, Inc. Apparatus and method for gathering the interest of a driver of an advertisement by tracking the sight of the driver
WO2015027241A1 (en) * 2013-08-23 2015-02-26 Tobii Technology Ab Systems and methods for providing audio to a user based on gaze input
US9143880B2 (en) * 2013-08-23 2015-09-22 Tobii Ab Systems and methods for providing audio to a user based on gaze input
US9196239B1 (en) * 2013-08-30 2015-11-24 Amazon Technologies, Inc. Distracted browsing modes
CN105683812B (en) * 2013-08-30 2018-07-03 英特尔公司 The incidence of nausea and detecting a head-mounted display, predict and mitigate
WO2018064141A1 (en) 2016-09-27 2018-04-05 Tobii Ab Portable eye tracking device
US20170017299A1 (en) * 2013-09-03 2017-01-19 Tobii Ab Portable eye tracking device
US9727136B2 (en) 2014-05-19 2017-08-08 Microsoft Technology Licensing, Llc Gaze detection calibration

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6320610B1 (en) * 1998-12-31 2001-11-20 Sensar, Inc. Compact imaging device incorporating rotatably mounted cameras
US20050175218A1 (en) * 2003-11-14 2005-08-11 Roel Vertegaal Method and apparatus for calibration-free eye tracking using multiple glints or surface reflections
US20100110368A1 (en) * 2008-11-02 2010-05-06 David Chaum System and apparatus for eyeglass appliance platform
US20110069277A1 (en) * 2009-04-01 2011-03-24 Tobii Technology Ab Visual display with illuminators for gaze tracking
US20130314303A1 (en) * 2010-02-28 2013-11-28 Osterhout Group, Inc. Ar glasses with user action control of and between internal and external applications with feedback
US20110211056A1 (en) * 2010-03-01 2011-09-01 Eye-Com Corporation Systems and methods for spatially controlled scene illumination
US20130021373A1 (en) * 2011-07-22 2013-01-24 Vaught Benjamin I Automatic Text Scrolling On A Head-Mounted Display
US20130050070A1 (en) * 2011-08-29 2013-02-28 John R. Lewis Gaze detection in a see-through, near-eye, mixed reality display
US20130114043A1 (en) * 2011-11-04 2013-05-09 Alexandru O. Balan See-through display brightness control
US20130114850A1 (en) * 2011-11-07 2013-05-09 Eye-Com Corporation Systems and methods for high-resolution gaze tracking
US20130300648A1 (en) * 2012-05-11 2013-11-14 Qualcomm Incorporated Audio user interaction recognition and application interface
US20150061996A1 (en) * 2013-09-03 2015-03-05 Tobii Technology Ab Portable eye tracking device
US9474131B2 (en) * 2014-05-02 2016-10-18 Lg Electronics Inc. Lighting device, lighting system and wearable device having image processor

Cited By (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9596391B2 (en) 2013-09-03 2017-03-14 Tobii Ab Gaze based directional microphone
US10116846B2 (en) 2013-09-03 2018-10-30 Tobii Ab Gaze based directional microphone
US9665172B2 (en) 2013-09-03 2017-05-30 Tobii Ab Portable eye tracking device
US9972132B2 (en) 2014-04-18 2018-05-15 Magic Leap, Inc. Utilizing image based light solutions for augmented or virtual reality
US10115232B2 (en) 2014-04-18 2018-10-30 Magic Leap, Inc. Using a map of the world for augmented or virtual reality systems
US10109108B2 (en) 2014-04-18 2018-10-23 Magic Leap, Inc. Finding new points by render rather than search in augmented or virtual reality systems
US10043312B2 (en) 2014-04-18 2018-08-07 Magic Leap, Inc. Rendering techniques to find new map points in augmented or virtual reality systems
US10115233B2 (en) 2014-04-18 2018-10-30 Magic Leap, Inc. Methods and systems for mapping virtual objects in an augmented or virtual reality system
US10013806B2 (en) 2014-04-18 2018-07-03 Magic Leap, Inc. Ambient light compensation for augmented or virtual reality
US10008038B2 (en) 2014-04-18 2018-06-26 Magic Leap, Inc. Utilizing totems for augmented or virtual reality systems
US20150316982A1 (en) * 2014-04-18 2015-11-05 Magic Leap, Inc. Utilizing pseudo-random patterns for eye tracking in augmented or virtual reality systems
US9996977B2 (en) 2014-04-18 2018-06-12 Magic Leap, Inc. Compensating for ambient light in augmented or virtual reality systems
US9761055B2 (en) 2014-04-18 2017-09-12 Magic Leap, Inc. Using object recognizers in an augmented or virtual reality system
US9767616B2 (en) 2014-04-18 2017-09-19 Magic Leap, Inc. Recognizing objects in a passable world model in an augmented or virtual reality system
US9766703B2 (en) 2014-04-18 2017-09-19 Magic Leap, Inc. Triangulation of points using known points in augmented or virtual reality systems
US9984506B2 (en) 2014-04-18 2018-05-29 Magic Leap, Inc. Stress reduction in geometric maps of passable world model in augmented or virtual reality systems
US9852548B2 (en) 2014-04-18 2017-12-26 Magic Leap, Inc. Systems and methods for generating sound wavefronts in augmented or virtual reality systems
US9881420B2 (en) 2014-04-18 2018-01-30 Magic Leap, Inc. Inferential avatar rendering techniques in augmented or virtual reality systems
US9911233B2 (en) 2014-04-18 2018-03-06 Magic Leap, Inc. Systems and methods for using image based light solutions for augmented or virtual reality
US9911234B2 (en) 2014-04-18 2018-03-06 Magic Leap, Inc. User interface rendering in augmented or virtual reality systems
US9922462B2 (en) 2014-04-18 2018-03-20 Magic Leap, Inc. Interacting with totems in augmented or virtual reality systems
US9928654B2 (en) * 2014-04-18 2018-03-27 Magic Leap, Inc. Utilizing pseudo-random patterns for eye tracking in augmented or virtual reality systems
US20150301599A1 (en) * 2014-04-18 2015-10-22 Magic Leap, Inc. Eye tracking systems and method for augmented or virtual reality
US10127723B2 (en) 2014-04-18 2018-11-13 Magic Leap, Inc. Room based sensors in an augmented reality system
US20150319826A1 (en) * 2014-05-02 2015-11-05 Lg Electronics Inc. Lighting system and control method thereof
US9655212B2 (en) * 2014-05-02 2017-05-16 Lg Electronics Inc. Lighting system having a plurality of lighting devices and an integrated control module
US9474131B2 (en) * 2014-05-02 2016-10-18 Lg Electronics Inc. Lighting device, lighting system and wearable device having image processor
US20150319827A1 (en) * 2014-05-02 2015-11-05 Lg Electronics Inc. Lighting system and control method thereof
US9727136B2 (en) * 2014-05-19 2017-08-08 Microsoft Technology Licensing, Llc Gaze detection calibration
US9952883B2 (en) 2014-08-05 2018-04-24 Tobii Ab Dynamic determination of hardware
US9844119B2 (en) * 2015-02-10 2017-12-12 Daqri, Llc Dynamic lighting for head mounted device
US20160231573A1 (en) * 2015-02-10 2016-08-11 Daqri, Llc Dynamic lighting for head mounted device
WO2017053871A3 (en) * 2015-09-24 2017-05-04 Supereye, Inc. Methods and devices for providing enhanced visual acuity
CN105676456A (en) * 2016-04-06 2016-06-15 众景视界(北京)科技有限公司 Modularized head-mounted electronic device

Also Published As

Publication number Publication date Type
CN108209857A (en) 2018-06-29 application
US20170272627A1 (en) 2017-09-21 application
KR20160111018A (en) 2016-09-23 application
CN105960193A (en) 2016-09-21 application
US9710058B2 (en) 2017-07-18 grant
US20150062322A1 (en) 2015-03-05 application
US9665172B2 (en) 2017-05-30 grant
WO2015034560A1 (en) 2015-03-12 application
CN105682539A (en) 2016-06-15 application
US20180020137A1 (en) 2018-01-18 application
EP3041401A1 (en) 2016-07-13 application
US20150061996A1 (en) 2015-03-05 application
CN105682539B (en) 2018-03-16 grant
US20180131850A1 (en) 2018-05-10 application
KR20160111019A (en) 2016-09-23 application
US10116846B2 (en) 2018-10-30 grant
KR101882594B1 (en) 2018-07-26 grant
US20150061995A1 (en) 2015-03-05 application
EP3041400A1 (en) 2016-07-13 application
US9596391B2 (en) 2017-03-14 grant
US20160328016A1 (en) 2016-11-10 application
US20150063603A1 (en) 2015-03-05 application
US9041787B2 (en) 2015-05-26 grant
WO2015034561A1 (en) 2015-03-12 application
KR20180086511A (en) 2018-07-31 application

Similar Documents

Publication Publication Date Title
US20130114850A1 (en) Systems and methods for high-resolution gaze tracking
US20100295769A1 (en) Device for Controlling an External Unit
US20130326364A1 (en) Position relative hologram interactions
US20140147829A1 (en) Wearable food nutrition feedback system
US20130241805A1 (en) Using Convergence Angle to Select Among Different UI Elements
US20120154277A1 (en) Optimized focal area for augmented reality displays
US8487838B2 (en) Gaze detection in a see-through, near-eye, mixed reality display
US20130242262A1 (en) Enhanced optical and perceptual digital eyewear
US20130342571A1 (en) Mixed reality system learned input and functions
US20120105473A1 (en) Low-latency fusing of virtual and real content
US8611015B2 (en) User interface
US8767306B1 (en) Display system
US20130050642A1 (en) Aligning inter-pupillary distance in a near-eye display system
US20140152558A1 (en) Direct hologram manipulation using imu
US9033502B2 (en) Optical measuring device and method for capturing at least one parameter of at least one eye wherein an illumination characteristic is adjustable
US8752963B2 (en) See-through display brightness control
US20100079449A1 (en) System and method for rendering dynamic three-dimensional appearing imagery on a two-dimensional user interface
US8971570B1 (en) Dual LED usage for glint detection
US9019174B2 (en) Wearable emotion detection and feedback system
US20160189426A1 (en) Virtual representations of real-world objects
US20140085446A1 (en) Optical device for the visually impaired
US20130050258A1 (en) Portals: Registered Objects As Virtualized, Personalized Displays
US8743051B1 (en) Mirror detection-based device functionality
US9171198B1 (en) Image capture technique
WO2015109145A1 (en) See-through computer display systems

Legal Events

Date Code Title Description
AS Assignment

Owner name: TOBII AB, SWEDEN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GUSTAFSSON, SIMON;CEDERLUND, MARKUS;KINGBAECK, ANDERS;SIGNING DATES FROM 20151217 TO 20151218;REEL/FRAME:037402/0721