WO2018156912A1 - Système d'interaction de regard - Google Patents

Système d'interaction de regard Download PDF

Info

Publication number
WO2018156912A1
WO2018156912A1 PCT/US2018/019447 US2018019447W WO2018156912A1 WO 2018156912 A1 WO2018156912 A1 WO 2018156912A1 US 2018019447 W US2018019447 W US 2018019447W WO 2018156912 A1 WO2018156912 A1 WO 2018156912A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
portable device
gaze
operating system
information
Prior art date
Application number
PCT/US2018/019447
Other languages
English (en)
Inventor
George-Svahn ERLAND
Original Assignee
Tobii Ab
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/444,035 external-priority patent/US20170235360A1/en
Application filed by Tobii Ab filed Critical Tobii Ab
Publication of WO2018156912A1 publication Critical patent/WO2018156912A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements

Definitions

  • the invention generally relates to computer implemented systems and methods for utilizing detection of eye movements for gaze driven interaction in connection with interactive graphical user interfaces, and in particular, to systems and methods for gaze interaction with portable devices. Further, the present invention relates to systems and methods for assisting a user when interacting with a graphical user interface by combining eye based input with gesture based input and gesture based user commands.
  • GUI graphical user interface
  • the interaction possibilities between the user and the different software applications run on the computer can be significantly enhanced.
  • one interesting idea for improving and facilitating the user interaction and for removing the bandwidth asymmetry is to use eye gaze tracking instead or as a complement to mouse input.
  • the cursor is positioned on the display according to the calculated point of gaze of the user.
  • a number of different techniques have been developed to select and activate a target object in these systems.
  • the system activates an object upon detection that the user fixates his or her gaze at a certain object for a certain period of time.
  • Another approach is to detect an activation of an object when the user's eye blinks.
  • a user looks at a target object, for example, a button on a graphical user interface and then presses a selection key of the keyboard. Once the selection key is pressed, a most probable target is determined using probability reasoning. The determined target object is then highlighted and the user can select it by pressing the selection key again. If the highlighted object is not the target object, the user can select another target object using additional keys to navigate to the intended target object.
  • a target object for example, a button on a graphical user interface and then presses a selection key of the keyboard. Once the selection key is pressed, a most probable target is determined using probability reasoning. The determined target object is then highlighted and the user can select it by pressing the selection key again. If the highlighted object is not the target object, the user can select another target object using additional keys to navigate to the intended target object.
  • the computer monitors an input device, such as a mouse, for mechanical activation by the operator.
  • an input device such as a mouse
  • the computer detects mechanical activation of the input device, it determined an initial cursor display position within a current gaze area.
  • the cursor is then displayed on the screen at the initial display position and thereafter the cursor is positioned manually according to the user's handling of the input device without regard to the gaze.
  • these portable devices function using touch as the primary or often only input method. This presents certain issues in ergonomics as well as usability. For example, when touching a screen on a mobile telephone/tablet, part of the screen is obscured. Further it may be difficult to touch the screen while simultaneously holding the phone/tablet, therefore two hands may be needed.
  • An object of the present invention is to provide improved methods, devices and systems for assisting a user when interacting with a graphical user interface by combining gaze based input with gesture based user commands.
  • Another object of the present invention is to provide methods, devices and systems for user friendly and intuitive interaction with graphical user interfaces.
  • a particular object of the present invention is to provide systems, devices and methods that enable a user of a computer system without a traditional touch-screen to interact with graphical user interfaces in a touch-screen like manner using a combination of gaze based input and gesture based user commands. Furthermore, the present invention offers a solution for touch-screen like interaction using gaze input and gesture based input as a complement or an alternative to touchscreen interactions with a computer device having a touch-screen, such as for instance in situations where interaction with the regular touch-screen is cumbersome or ergonomically challenging.
  • Another particular object of the present invention is to provide systems, devices and methods for combined gaze and gesture based interaction with graphical user interfaces to achieve a touchscreen like environment in computer systems without a traditional touchscreen or in computer systems having a touchscreen arranged economically unfavorable for the user or a touchscreen arranged such that it is more comfortable for the user to use gesture and gaze for the interaction than the touchscreen.
  • GUI Graphic User Interface
  • object or “object part” refer to an interactive graphical object or GUI object such as a window, an icon, a button, a scroll bar, a hyperlink, or non-interactive objects such as an image, text or a word in a text that the user desires to select or activate.
  • touchpad refers to a surface sensor for detecting the position and movement of one or multiple fingers and/or one or multiple other objects intended for pointing, drawing or making gestures, such as for instance a stylus.
  • control module for implementation in, for example, a computer device or handheld device or a wireless
  • the control module is configured to acquire user input from input means adapted to detect user generated gestures and gaze data signals from a gaze tracking module and to determine at least one user generated gesture based control command based on the user input. Further, the control module is configured to determine a gaze point area on the information presentation area including the user's gaze point based on at least the gaze data signals and to execute at least one user action manipulating a view presented on the graphical information presentation area based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • the gaze point area serving as a starting point may be an area at which the user initially gazes at or a fine tuned area, i.e. an area that the user has selected by tuning or correcting commands via, for example, the input means, thereby correcting or tuning an initial gaze point area to a selected area.
  • a method for generating gesture based commands during user interaction with an information presentation area for example, associated with or included in a computer device or handheld device, or associated with or included in a wireless transmit/receive unit (WTRU).
  • the method comprises acquiring user input corresponding to user generated gestures and gaze data signals and determining at least one user generated gesture based control command based on the user input.
  • a gaze point area on the information presentation area including the user's gaze point is determined based on at least the gaze data signals and at least one user action manipulating a view presented on the information presentation area is executed based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • a handheld portable device provided with or associated with an information presentation area and comprising input means adapted to detect user generated gestures and a gaze tracking module adapted to detect gaze data of a viewer of the information presentation area.
  • the handheld device further comprises a control module configured to acquire user input from the input means and gaze data signals from the gaze tracking module and to determine at least one user generated gesture based control command based on the user input.
  • the control module is further configured to determine a gaze point area on the information presentation area including the user's gaze point based on at least the gaze data signals and to execute at least one user action manipulating a view presented on the information presentation area based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • the handheld device may be a cellular phone, a smartphone, an iPad or similar device, a tablet, a phoblet/phablet, a laptop or similar device.
  • a wireless transmit/receive unit associated with an information presentation area and comprising input means adapted to detect user generated gestures and a gaze tracking module adapted to detect gaze data of a viewer of the information presentation area.
  • the WTRU further comprises a control module configured to acquire user input from the input means and gaze data signals from the gaze tracking module and to determine at least one user generated gesture based control command based on the user input.
  • the control module is further configured to determine a gaze point area on the information presentation area including the user's gaze point based on at least the gaze data signals and to execute at least one user action manipulating a view presented on the information presentation area based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • wireless transmit/receive unit include but is not limited to a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a cellular telephone, a smartphone, a personal digital assistant (PDA), a computer, or any other type of device capable of operating in a wireless environment such as a wireless local area network (WLAN) or wireless mobile communication system (e.g. a third generation (3G) global system for mobile
  • UE user equipment
  • PDA personal digital assistant
  • a computer or any other type of device capable of operating in a wireless environment such as a wireless local area network (WLAN) or wireless mobile communication system (e.g. a third generation (3G) global system for mobile
  • WLAN wireless local area network
  • 3G third generation
  • LTE long term evolution
  • a system for user interaction with an information presentation area comprising input means adapted to detect user generated gestures and a gaze tracking module adapted to detect gaze data of a viewer of the information presentation area. Further, the system includes a control module configured to acquire user input from the input means and gaze data signals from the gaze tracking module and to determine at least one user generated gesture based control command based on the user input.
  • the control module is further configured to determine a gaze point area on the information presentation area where the user's gaze point is located based on at least the gaze data signals and to execute at least one user action manipulating a view presented on the graphical information presentation area based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • a computer device associated with an information presentation area.
  • the computer device comprises input means adapted to detect user generated gestures and a gaze tracking module adapted to detect gaze data of a viewer of the information presentation area.
  • the computer device further comprises a control module configured to acquire user input from input means adapted to detect user generated gestures and gaze data signals from a gaze tracking module and to determine at least one user generated gesture based control command based on the user input.
  • control module is configured to determine a gaze point area on the information presentation area including the user's gaze point based on at least the gaze data signals and to execute at least one user action manipulating a view presented on the information presentation area based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • the computer device may, for example, be any one from the group of a personal computer, computer workstation, mainframe computer, a processor or device in a vehicle, or a handheld device such as a cell phone, smartphone or similar device, portable music player (such as e.g. an iPod), laptop computers, computer games, electronic books, an iPAD or similar device, a Tablet, a Phoblet/Phablet.
  • a personal computer computer workstation, mainframe computer, a processor or device in a vehicle, or a handheld device such as a cell phone, smartphone or similar device, portable music player (such as e.g. an iPod), laptop computers, computer games, electronic books, an iPAD or similar device, a Tablet, a Phoblet/Phablet.
  • a handheld device such as a cell phone, smartphone or similar device, portable music player (such as e.g. an iPod), laptop computers, computer games, electronic books, an iPAD or similar device, a Tablet, a Phoblet/P
  • the input means is configured to detect user gestures by a hand or a finger (or fingers), for example, relative a keyboard or an information presentation area using, for example, an optical measurement technique or capacitive measurement technique.
  • a system for user interaction with a wearable head mounted information presentation area comprising input means configured as a gyro ring adapted to detect user generated gestures and adapted to wirelessly communicate with a control module also communicatively connected to the information presentation area as well as a gaze tracking module adapted to detect gaze data of a viewer of the information presentation area.
  • a control module configured to: acquire user input from the input means and gaze data signals from the gaze tracking module; determine at least one user generated gesture based control command based on the user input; determine a gaze point area on the information presentation area including the user's gaze point based on at least the gaze data signals; and execute at least one user action manipulating a view presented on the graphical information presentation area based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • a system for user interaction with an information presentation area comprises input means adapted to detect user generated gestures, wherein the input means comprising at least one touchpad arranged on a steering device of a vehicle or adapted to be integrated in a steering device of a vehicle.
  • the system comprises a gaze tracking module adapted to detect gaze data of a viewer of the information presentation area and a control module configured to: acquire user input from the input means and gaze data signals from the gaze tracking module; determine at least one user generated gesture based control command based on the user input; determine a gaze point area on the information presentation area including the user's gaze point based on at least the gaze data signals; and execute at least one user action manipulating a view presented on the graphical information presentation area based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • the input means includes a touchpad configured to enable a user to generate gesture based control commands.
  • the gesture based commands can for example be generated by moving at least one finger over a surface of the touchpad or touching a surface of the touchpad with, for example, the finger.
  • a dedicated part or area of the touchpad surface is configured to receive gesture based control commands.
  • At least a first dedicated part or area of the touchpad surface is configured to receive a first set of gesture based control commands and at least a second part or area of the touchpad surface is configured to receive a second set of gesture based control commands.
  • the touchpad may be configured to receive gestures such as scrolling or zooming at a dedicated area or part.
  • the control module is configured to determine at least one gesture based control command based on multiple simultaneous user input via the input means. Further, a gaze point area on the information presentation area where the user's gaze point is located is determined based on the gaze data signals and at least one user action manipulating a view presented on the graphical information presentation area is executed based on the determined gaze point area and the at least one gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • an input module is configured to interpret signals representing at least one user generated gesture to provide at least one gesture based control command reflecting a user's gesture.
  • the input module is arranged in the control module.
  • the input module is configured to interpret the signals representing the at least one user generated gesture using gaze input signals and/or a predetermined set of possible gesture based control commands, each possible control command corresponding to a particular user gesture relative the input means.
  • At least one object is presented on the graphical information presentation area, the object representing at least one graphical user interface component and configured to be manipulated based on the user-generated gesture based control commands, wherein the control module is configured to determine if the gaze point of the user is on an object or in an area surrounding that object based on the gaze data signals. Further, the control module may be configured to determine if the gaze point of the user has been on an object or in an area surrounding that object at a predetermined point in time based on the gaze data signals. For example, the control module may be configured to determine if the gaze point of the user was on an object or the area surrounding that object 0.1 seconds ago.
  • User activation of the object is enabled if the user's gaze point is on or within an area surrounding that object synchronized with a user generated activation command resulting from user input via the input means, wherein the activated object can be manipulated by user generated commands resulting from user input via the input means.
  • User activation of the object may also be enabled if the user's gaze point was on or within an area surrounding that object at the
  • the location of the initial gaze point is indicated by a visual feedback, such as a crosshairs or similar sign.
  • the user may adjust this initial location by moving the finger on the touchpad. Then, the user may, in a touchscreen like manner, interact with the information presentation area using different gestures.
  • the strength of the visual feedback e.g. the strength of the light of a crosshairs, may be dependent on where the user's gaze is located on the information presentation area. For example, if a dragging operation to pan a window is initiated at the gaze point, the visual feedback may initially be discrete. When the dragging operation has been maintained for a period, the visual feedback can be strengthened to indicate for the user where the dragging operation is performed at the moment.
  • the gestures are finger movements relative the touchpad and each gesture is associated with or corresponds to a particular gesture based control command resulting in a user action.
  • a visual feedback related to that object is presented. For example, by pressing down and holding the finger on the touchpad during a first period of time, the object may be highlighted and, by continue to hold the finger on the touchpad for a second period of time, an information box presenting information regarding the object may be displayed.
  • a primary action By gazing, for example, at an object presented on the information presentation area and by in connection to this tapping on the touchpad using a finger, a primary action can be initiated.
  • an application can be opened and started by gazing at an icon representing the application and tapping on the touchpad using a finger.
  • a primary action By gazing, for example, at an object presented on the information presentation area and by, in connection to this, lifting a finger (or fingers) that have been in contact with the touchpad, a primary action can be initiated.
  • an application can be opened and started by gazing at an icon representing the application and lifting a finger (or fingers) that have been in contact with the touchpad.
  • the user may slide or drag the view presented by the information presentation area by gazing at the information presentation area and by, in connection to this, sliding his or her finger over the touchpad. The dragging is then initiated at the gaze point of the user.
  • a similar action to slide an object over the information presentation area can be achieved by gazing at the object and by, in connection to this, sliding the finger over the touchpad. Both of these objectives may instead be implemented in a way where two fingers are required to do the swipe, or one finger is used for swiping while another finger holds down a button.
  • the user may select an object for further actions by gazing at the object and by, in connection to this, swiping his or her finger downwards on the touchpad.
  • a menu or other window hidden during normal use such as a help menu
  • a hidden menu or other window can be displayed or presented if the user gazes at, for example, the left edge of the information presentation area and swipes his or her finger over the touchpad in the right direction.
  • the finger By gazing at a slider control, for example a volume control, the finger can be moved up/down (or left/right for a horizontal control) on the touch pad, on a predefined area of a touch screen or above a keyboard to adjust the value of the slider control.
  • a slider control for example a volume control
  • the different options can be displayed on different sides of the object after a preset focusing dwell time has passed or after appropriate user input has been provided.
  • the touchpad or a predefined area of a touch screen is thereafter used to choose action. For example, slide left to copy and slide right to rename.
  • the gaze tracking module and the user input means are implemented in a touchscreen provided device such as an iPad or similar device.
  • the touchscreen functions both as information presentation area and input device for input of user gestures.
  • a control module is included in the touchscreen provided device and is configured to determine a gaze point area on the information presentation area, i.e. the
  • the user's gaze point is located based on the gaze data signals and to execute at least one user action manipulating a view presented on the touchscreen based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • the user gestures are inputted via the touchscreen.
  • the user gestures, or finger movements on the touchscreen are relative to the gaze point, which entails a more user friendly and ergonomic use of touchscreen provided devices.
  • the user may hold the device with both hands and interact with graphical user interfaces on the touchscreen using the gaze and movement of the thumbs, where all user actions and activations have the gaze point of the user as starting point.
  • the gesture and gaze initiated actions discussed above are only exemplary and there are a large number of further gestures in combination with gaze point resulting in an action that are conceivable. Below, some further examples are described:
  • Selection of an object or object part can be made by gazing at that object or object part and pressing a finger (e.g. a thumb), fine tuning by moving the finger and releasing the pressure applied by the finger to select that object or object part;
  • a finger e.g. a thumb
  • Selection of an object or object part can be made by gazing at that object or object part, pressing a finger (e.g. a thumb), fine tuning by moving the finger, using another finger (e.g. the other thumb) to tap for selecting that object or object part.
  • a finger e.g. a thumb
  • fine tuning by moving the finger, using another finger (e.g. the other thumb) to tap for selecting that object or object part.
  • a double tap may be used for a "double click action” and a quick downward movement may be used for a "right click”.
  • an automatic panning function can be activated so that the presentation area is continuously slid from one of the edges of the screen towards the center while the gaze point is near the edge of the information presentation area, until a second user input is received.
  • the presentation area is instantly slid according to the gaze point (e.g., the gaze point is used to indicate the center of where the information presentation area should be slid).
  • one of the fingers can be used to fine-tune the point of action.
  • a user feedback symbol like a "virtual finger” can be shown on the gaze point when the user touches the touchscreen.
  • the first finger can be used to slide around to adjust the point of action relative to the original point.
  • the point of action is fixed and the second finger is used for "clicking" on the point of action or for performing two-finger gestures like the rotate, drag and zoom examples above.
  • the gaze tracking module and the user input means are implemented in a portable device such as an iPad, ultrabook tablet or similar device.
  • a portable device such as an iPad, ultrabook tablet or similar device.
  • one or two separate touchpads are placed on the back side of the device to allow two-finger gestures with other fingers than the thumb.
  • the gaze tracking module and the user input means are implemented in a vehicle.
  • the information presentation area may be a heads-up display or an infotainment screen.
  • the input means may be one or two separate touch pads on the backside (for use with the index finger/s) or on the front side (for use with the thumb/s) of the steering wheel.
  • the gaze tracking module and the information presentation area are implemented in a wearable head mounted display that may be designed to look as a pair of glasses (such as the solution described in U.S. Pat. No. 8,235,529).
  • the user input means may include a gyro and be adapted to be worn on a wrist, hand or at least one finger.
  • the input means may be a ring with a wireless connection to the glasses (or to a processing unit such as a smart phone that is communicatively connected to the glasses) and a gyro that detects small movements of the finger where the ring is worn.
  • the detected movements representing gesture data may then wirelessly be communicated to the glasses where gaze is detected and gesture based control commands based on the gesture data from the input means is used to identify and execute user action.
  • the touchpad is significantly smaller than the information presentation area, which entails that in certain situations the touchpad may impose limitations on the possible user actions. For example, it may be desired to drag or move an object over the entire information presentation area while the user's movement of a finger or fingers is limited by the smaller touchpad area. Therefore, in embodiments of the present invention, a touchscreen like session can be maintained despite that the user has removed the finger or fingers from the touchpad if, for example, a specific or dedicated button or keyboard key is held down or pressed. Thereby, it is possible for the user to perform actions requiring multiple touches on the touchpad. For example, an object can be moved or dragged across the entire information presentation area by means of multiple dragging movements on the touchpad.
  • a dragging movement on the information presentation area or other user action is continued after the finger or fingers has reached an edge of the touchpad in the same direction as the initial direction of the finger or fingers.
  • the continued movement or other actions may be continued until an interruption command is delivered, which may be, for example, a pressing down of a keyboard key or button, a tap on the touchpad or when the finger or fingers is removed from the touchpad.
  • the speed of the dragging movement or other action is increased or accelerated when the user's finger or fingers approaches the edge of the touchpad.
  • the speed may be decreased if the fingers or finger is moved in an opposite direction.
  • the action e.g. a dragging movement of an object
  • the action can be accelerated based on gaze position. For example, by gazing at an object, initiating a dragging operation of that object in a desired direction and thereafter gazing at a desired end position for that object, the speed of the object movement will be higher the longer the distance between the initial position of the object and the desired end position is.
  • voice commands may be used to choose what action to perform on the object currently being gazed at and then a gesture is required to fulfill the action.
  • a voice command such as the word "move” may allow the user to move the object currently being gazed at by moving a finger over the touchpad or touchscreen.
  • Another action to perform may be to delete an object.
  • the word "delete” may allow deletion of the object currently being gazed at, but additionally a gesture, such as swiping downwards is required to actually delete the object.
  • the object to act on is chosen by gazing at it, the specific action to perform is chosen by a voice command and the movement to perform or the confirmation is done by a gesture.
  • Another object of the present invention is to provide systems and methods which provide for convenient interaction with a portable device.
  • any reference to “portable device”, “mobile device” or similar is intended to refer to a computing device that may be carried by a user. This includes, but is not limited to, mobile telephones, tablets, laptops and virtual reality headsets.
  • the present invention relates to the following:
  • the gaze of a user may be determined using an eye tracking device or components operatively connected with the portable device.
  • the components of the eye tracking device may be integrated into the portable device.
  • a typical eye tracking device comprises an image sensor and at least one illuminator, preferably an infrared illuminator, and the image sensor captures an image of at least one eye of the user. Reflections caused by the illuminator or the illuminators may be extracted from the captured image and compared with a feature of the eye in order to determine the user's gaze direction.
  • an illuminator may not be present and merely ambient light used. Any other eye tracking device may also function with the present invention, the concept of eye tracking is not the object of the present invention.
  • any reference to information displayed on a portable device is intended to represent the entire range of information that may be displayed on a display, this includes text, images, video, icons and the like.
  • FIG. 1 shows an overview picture of a user controlling a computer apparatus in which the present invention is implemented
  • FIG. 2 is a block diagram illustrating an embodiment of an arrangement in accordance with the present invention.
  • FIG. 3 is a block diagram illustrating another embodiment of an arrangement in accordance with the present invention.
  • FIG. 4 illustrates an exemplary gesture resulting in a user generated gesture based control command in accordance with the present invention
  • FIG. 5 illustrates another exemplary gesture resulting in a user generated gesture based control command in accordance with the present invention
  • FIG. 6 illustrates a further exemplary gesture resulting in a user generated gesture based control command in accordance with the present invention
  • FIG. 7 illustrates yet another exemplary gesture resulting in a user generated gesture based control command in accordance with the present invention
  • FIG. 8 illustrates a further exemplary gesture resulting in a user generated gesture based control command in accordance with the present invention
  • FIG. 9 illustrates another exemplary gesture resulting in a user generated gesture based control command in accordance with the present invention.
  • FIG. 10 illustrates yet another exemplary gesture resulting in a user generated gesture based control command in accordance with the present invention
  • FIG. 11a shows an overview picture of a touchscreen provided device in which a further embodiment of the present invention is implemented
  • FIG. 1 lb shows an overview picture of a device provided with touchpads on a backside in which a further embodiment of the present invention is implemented;
  • FIG. 12 is a block diagram illustrating the embodiment in accordance with the present invention shown in FIG. 11a;
  • FIG. 13a is a schematic view of a control module according to an embodiment of the present invention.
  • FIG. 13b is a schematic view of a control module according to another embodiment of the present invention.
  • FIG. 13c is a schematic view of a control module according to another embodiment of the present invention.
  • FIG. 14 is a schematic view of a wireless transmit/receive unit, WTRU, according to an embodiment of the present invention.
  • FIG. 15a is a schematic view of an embodiment of a computer device or handheld device in accordance with an embodiment of the present invention.
  • FIG. 15b is a schematic view of another embodiment of a computer device or handheld device in accordance with the present invention.
  • FIG. 16 is a schematic flow chart illustrating steps of an embodiment of a method in accordance with an embodiment of the present invention.
  • FIG. 17 is a schematic flow chart illustrating steps of another embodiment of a method in accordance with the present invention.
  • FIG. 18 is a schematic flow chart illustrating steps of a further embodiment of a method in accordance with an embodiment of the present invention.
  • FIG. 19 is a schematic flow chart illustrating steps of another embodiment of a method in accordance with an embodiment of the present invention.
  • FIG. 20 is a block diagram illustrating a further embodiment of an arrangement in accordance with the present invention.
  • FIG. 21 is a schematic illustration of yet another implementation of the present invention.
  • FIG. 22 is a schematic illustration of a further implementation of the present invention.
  • FIG. 23 is a schematic illustration of an implementation of the present invention.
  • FIG. 24 shows a block diagram of one method of the invention for interacting with a portable device.
  • FIG. 25 shows a chart of a method of transitioning between visual indications on a display, when a gaze signal is momentarily lost.
  • module refers to an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that execute one or more software programs, a combinational logic circuit, or other suitable components that provide the described functionality.
  • ASIC application specific integrated circuit
  • module further refers to a specific form of software necessary to practice the methods described herein and particularly the functions described in connection with each specific “module”. It is believed that the particular form of software will be determined primarily by the particular system architecture employed in the system and by the particular methodologies employed by the system according to the present invention.
  • FIG. 1 shows an embodiment of a computer system with integrated gaze and manual control according to the present invention.
  • the user 110 is able to control the computer system 10 at least partly based on an eye-tracking signal D.sub.EYE, which described the user's point of regard x, y on an information presentation area or display 20 and based on user generated gestures, i.e. a movement of at least one body part of the user can be detected, generating gesture based control commands via user input means 50 such as a touchpad 51.
  • D.sub.EYE eye-tracking signal
  • user generated gestures i.e. a movement of at least one body part of the user can be detected
  • user input means 50 such as a touchpad 51.
  • touchpad refers to a pointing device featuring a tactile sensor, a specialized surface that can translate the motion and position of a user's fingers to a relative position on a screen (information presentation area).
  • Touchpads are a common feature of laptop computers, and are also used as a substitute for a mouse where desk space is scarce. Because they vary in size, they can also be found on personal digital assistants (PDAs) and some portable media players. Wireless touchpads are also available as detached accessories. Touchpads operate in one of several ways, including capacitive sensing and conductance sensing.
  • Some touchpads and associated device driver software may interpret tapping the pad as a click, and a tap followed by a continuous pointing motion (a "click-and-a-half ') can indicate dragging.
  • Tactile touchpads allow for clicking and dragging by incorporating button functionality into the surface of the touchpad itself. To select, one presses down on the touchpad instead of a physical button. To drag, instead performing the "click-and-a-half technique, one presses down while on the object, drags without releasing pressure and lets go when done.
  • Touchpad drivers can also allow the use of multiple fingers to facilitate the other mouse buttons (commonly two-finger tapping for the center button).
  • Some touchpads have "hotspots", locations on the touchpad used for functionality beyond a mouse.
  • touchpads For example, on certain touchpads, moving the finger along an edge of the touch pad will act as a scroll wheel, controlling the scrollbar and scrolling the window that has the focus vertically or horizontally. Apple uses two-finger dragging for scrolling on their trackpads. Also, some touchpad drivers support tap zones, regions where a tap will execute a function, for example, pausing a media player or launching an application. All of these functions are implemented in the touchpad device driver software, and can be disabled. Touchpads are primarily used in self-contained portable laptop computers and do not require a flat surface near the machine.
  • the touchpad is close to the keyboard, and only very short finger movements are required to move the cursor across the display screen; while advantageous, this also makes it possible for a user's thumb to move the mouse cursor accidentally while typing.
  • Touchpad functionality is available for desktop computers in keyboards with built-in touchpads.
  • touchpads examples include one-dimensional touchpads used as the primary control interface for menu navigation on second-generation and later iPod Classic portable music players, where they are referred to as "click wheels", since they only sense motion along one axis, which is wrapped around like a wheel.
  • the second-generation Microsoft Zune product line (the Zune 80/120 and Zune 4/8) uses touch for the Zune Pad.
  • Apple's PowerBook 500 series was its first laptop to carry such a device, which Apple refers to as a "trackpad”.
  • Apple's more recent laptops feature trackpads that can sense up to five fingers simultaneously, providing more options for input, such as the ability to bring up the context menu by tapping two fingers.
  • Apple's revisions of the MacBook and MacBook Pro incorporated a "Tactile Touchpad” design with button functionality incorporated into the tracking surface.
  • the present invention provides a solution enabling a user of a computer system without a traditional touchscreen to interact with graphical user interfaces in a touchscreen like manner using a combination of gaze based input and gesture based user commands. Furthermore, the present invention offers a solution for touchscreen like interaction using gaze input and gesture based input as a complement or an alternative to touchscreen interactions with a computer device having a touchscreen.
  • the display 20 may hence be any type of known computer screen or monitor, as well as combinations of two or more separate displays.
  • the display 20 may constitute a regular computer screen, a stereoscopic screen, a heads-up display (HUD) in a vehicle, or at least one head-mounted display (HMD).
  • HUD heads-up display
  • HMD head-mounted display
  • the computer 30 may, for example, be any one from the group of a personal computer, computer workstation, mainframe computer, a processor in a vehicle, or a handheld device such as a cell phone, portable music player (such as e.g. an iPod), laptop computers, computer games, electronic books and similar other devices.
  • the present invention may also be implemented in "intelligent environment" where, for example, objects presented on multiple displays can be selected and activated.
  • a gaze tracker unit 40 is included in the display 20, or is associated with the display 20.
  • a suitable gaze tracker is described in the U.S. Pat. No. 7,572,008, titled "Method and Installation for detecting and following an eye and the gaze direction thereof, by the same applicant, which hereby is incorporated in its entirety.
  • the software program or software implemented instructions associated with the gaze tracking module 40 may be included within the gaze tracking module 40.
  • the specific example shown in FIGS. 2, 3 and 20 illustrates the associated software implemented in a gaze tracking module, which may be included solely in the computer 30, in the gaze tracking module 40, or in a combination of the two, depending on the particular application.
  • the computer system 10 comprises a computer device 30, a gaze tracking module 40, a display 20, a control module 36, 36' and user input means 50, 50' as shown in FIGS. 2, 3 and 20.
  • the computer device 30 comprises several other components in addition to those illustrated in FIGS. 2 and 20 but these components are omitted from FIGS. 2, 3 and 20 in illustrative purposes.
  • the user input means 50, 50' comprises elements that are sensitive to pressure, physical contact, gestures, or other manual control by the user, for example, a touchpad 51.
  • the input device means 50, 50' may also include a computer keyboard, a mouse, a "track ball", or any other device, for example, an IR-sensor, voice activated input means, or a detection device of body gestures or proximity based input can be used.
  • a touchpad 51 is included in the user input device 50, 50'.
  • An input module 32 which may be a software module included solely in a control module 36' or in the user input means 50 or as a module separate from the control module and the input means 50', is configured to receive signals from the touchpad 51 reflecting a user's gestures.
  • the input module 32 is also adapted to interpret the received signals and provide, based on the interpreted signals, gesture based control commands, for example, a tap command to activate an object, a swipe command or a slide command.
  • the control module 36 includes the input module 32 based on gesture data from the user input means 50', see FIG. 3.
  • the control module 36, 36' is further configured to acquire gaze data signals from the gaze tracking module 40. Further, the control module 36, 36' is configured to determine a gaze point area 120 on the information presentation area 20 where the user's gaze point is located based on the gaze data signals.
  • the gaze point area 120 is preferably, as illustrated in FIG. 1, a local area around a gaze point of the user.
  • control module 36, 36' is configured to execute at least one user action manipulating a view presented on the graphical information presentation area 20 based on the determined gaze point area and the at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • the control module 36, 36' may be integrated in the computer device 30 or may be associated or coupled to the computer device 30.
  • the present invention allows a user to interact with a computer device 30 in touchscreen like manner, e.g. manipulate objects presented on the information presentation area 20, using gaze and gestures, e.g. by moving at least one finger on a touchpad 51.
  • the location of the initial gaze point is indicated by a visual feedback, such as a crosshairs or similar sign.
  • This initial location can be adjusted by moving the finger on the touchpad 51.
  • the user can, in a touchscreen like manner, interact with the information presentation area 20 using different gestures and the gaze.
  • the gestures are finger movements relative the touchpad 51 and each gesture is associated with or corresponds to particular gesture based user command resulting in a user action.
  • a primary action can be initiated.
  • an application can be opened and started by gazing at an icon representing the application and tapping on the touchpad 51 using a finger.
  • this gesture is illustrated in relation to a touchpad 51.
  • the user may slide or drag the view presented by the information presentation area 20 by gazing somewhere on the information presentation area 20 and by, in connection to this, sliding his or her finger 81 over the touchpad 51.
  • a similar action to slide an object over the information presentation area 20 can be achieved by gazing at the object and by, in connection to this, sliding the finger 81 over the touchpad 51.
  • This gesture is illustrated in FIG. 6 in relation to the touchpad 51.
  • this gesture can be executed by means of more than one finger, for example, by using two fingers.
  • the user may select an object for further actions by gazing at the object and by, in connection to this, swiping his or her finger 91 on the touchpad 51 in a specific direction.
  • This gesture is illustrated in FIG. 7 in relation to the touchpad 51.
  • this gesture can be executed by means of more than one finger, for example, by using two fingers.
  • the finger By gazing at a slider control, for example a volume control, the finger can be moved up/down (or left/right for a horizontal control) to adjust the value of the slider control.
  • this gesture can be detected on a touchpad, on a touch screen or in air without physically touching the input means.
  • a gesture is done to choose action. For example, swipe left to copy and swipe right to rename. With appropriate input means this gesture can be detected on a touchpad, on a touch screen or in air without physically touching the input means. By pressing the finger harder on the touchpad, i.e. increasing the pressure of a finger touching the touchpad, a sliding mode can be initiated.
  • the object can be moved or dragged over the information presentation area.
  • the touchscreen like session is finished.
  • the user may thereafter start a new touchscreen like session by gazing at the information presentation area 20 and placing the finger on the touchpad 51.
  • gesture and gaze initiated actions discussed above are only exemplary and there are a large number of further gestures in combination with gaze point resulting in an action that are conceivable.
  • appropriate input means many of these gestures can be detected on a touchpad, on a predefined area of a touch screen, in air without physically touching the input means, or by an input means worn on a finger or a hand of the user.
  • some further examples are described:
  • Selection of an object or object part can be made by gazing at that object or object part and pressing a finger (e.g. a thumb), fine tuning by moving the finger and releasing the pressure applied by the finger to select that object or object part;
  • a finger e.g. a thumb
  • Selection of an object or object part can be made by gazing at that object or object part, pressing a finger (e.g. a thumb), fine tuning by moving the finger, using another finger (e.g. the other thumb) to tap for selecting that object or object part.
  • a finger e.g. a thumb
  • fine tuning by moving the finger, using another finger (e.g. the other thumb) to tap for selecting that object or object part.
  • a double tap may be used for a "double click action” and a quick downward movement may be used for a "right click”.
  • an automatic panning function can be activated so that the presentation area is continuously slided from one of the edges of the screen towards the center while the gaze point is near the edge of the information presentation area, until a second user input is received.
  • the presentation area is instantly slid according to the gaze point (e.g., the gaze point is used to indicate the center of where the information presentation area should be slid).
  • one of the fingers can be used to fine-tune the point of action.
  • a user feedback symbol like a "virtual finger” can be shown on the gaze point when the user touches the touchscreen.
  • the first finger can be used to slide around to adjust the point of action relative to the original point.
  • the point of action is fixed and the second finger is used for "clicking" on the point of action or for performing two-finger gestures like the rotate, drag and zoom examples above.
  • the touchscreen like session can be maintained despite that the user has removed the finger or fingers from the touchpad if, for example, a specific or dedicated button or keyboard key is held down or pressed.
  • a specific or dedicated button or keyboard key is held down or pressed.
  • FIG. 11a shows a further embodiment of a system with integrated gaze and manual control according to the present invention.
  • This embodiment of the system is implemented in a device 100 with a touchscreen 151 such as an iPad or similar device.
  • the user is able to control the device 100 at least partly based on gaze tracking signals which describes the user's point of regard x, y on the touchscreen 151 and based on user generated gestures, i.e. a movement of at least one body part of the user can be detected, generating gesture based control commands via user input means 150 including the touchscreen 151.
  • the present invention provides a solution enabling a user of a device 100 with a touchscreen 151 to interact with a graphical user interfaces using gaze as direct input and gesture based user commands as relative input. Thereby, it is possible, for example, to hold the device 100 with both hands and interact with a graphical user interface 180 presented on the touchscreen with gaze and the thumbs 161 and 162 as shown in FIG. 11a.
  • one or more touchpads 168 can be arranged on the backside of the device 100', i.e. on the side of the device on which the user normally do not look at during use. This embodiment is illustrated in FIG. 1 lb.
  • a user is allowed to control the device at least partly based on gaze tracking signals which describes the user's point of regard x, y on the information presentation area and based on user generated gestures, i.e. a movement of at least one finger on the one or more touchpads 168 on the backside of the device 100', generating gesture based control commands interpreted by the control module.
  • a gaze tracking module 140 is included in the device 100, 100'.
  • a suitable gaze tracker is described in the U.S. Pat. No. 7,572,008, titled "Method and Installation for detecting and following an eye and the gaze direction thereof, by the same applicant, which hereby is incorporated in its entirety.
  • the software program or software implemented instructions associated with the gaze tracking module 140 may be included within the gaze tracking module 140.
  • the device 100 comprises a gaze tracking module 140, user input means 150 including the touchscreen 151 and an input module 132, and a control module 136 as shown in FIG. 12.
  • the device 100 comprises several other components in addition to those illustrated in FIG. 12 but these components are omitted from FIG. 12 in illustrative purposes.
  • the input module 132 which may be a software module included solely in a control module or in the user input means 150, is configured to receive signals from the touchscreen 151 reflecting a user's gestures. Further, the input module 132 is also adapted to interpret the received signals and provide, based on the interpreted signals, gesture based control commands, for example, a tap command to activate an object, a swipe command or a slide command.
  • the control module 136 is configured to acquire gaze data signals from the gaze tracking module 140 and gesture based control commands from the input module 132. Further, the control module 136 is configured to determine a gaze point area 180 on the information presentation area, i.e. the touchscreen 151, where the user's gaze point is located based on the gaze data signals.
  • the gaze point area 180 is preferably, as illustrated in FIG. 1, a local area around a gaze point of the user.
  • control module 136 is configured to execute at least one user action manipulating a view presented on the touchscreen 151 based on the determined gaze point area and the at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point. All user actions described in the context of this application may also be executed with this embodiment of the present invention.
  • the location of the initial gaze point is indicated by a visual feedback, such as a crosshairs or similar sign. This initial location can be adjusted by moving the finger on the touchscreen 151, for example, using a thumb 161 or 162.
  • the user can interact with the touchscreen 151 using different gestures and the gaze, where the gaze is the direct indicator of the user's interest and the gestures are relative to the touchscreen 151.
  • the gestures are finger movements relative the touchscreen 151 and each gesture is associated with or corresponds to particular gesture based user command resulting in a user action.
  • control modules for generating gesture based commands during user interaction with an information presentation area 201 for example, associated with a WTRU (described below with reference to FIG. 14), or a computer device or handheld portable device (described below with reference to FIG. 15a or 15b), or in a vehicle (described below with reference to FIG. 21), or in a wearable head mounted display (described below with reference to FIG. 22) will be described. Parts or modules described above will not be described in detail again in connection to this embodiment.
  • the control module 200 is configured to acquire user input from input means 205, for example, included in a device in which the control module may be arranged in, adapted to detect user generated gestures.
  • the control module 200 may include an input module 232 comprising a data acquisition module 210 configured to translate the gesture data from the input means 205 into an input signal.
  • the input means 205 may include elements that are sensitive to pressure, physical contact, gestures, or other manual control by the user, for example, a touchpad. Further, the input means 205 may also include a computer keyboard, a mouse, a "track ball", or any other device, for example, an IR-sensor, voice activated input means, or a detection device of body gestures or proximity based input can be used.
  • the input module 232 is configured to determine at least one user generated gesture based control command based on the input signal.
  • the input module 232 further comprises a gesture determining module 220 communicating with the data acquisition module 210.
  • the gesture determining module 220 may also communicate with the gaze data analyzing module 240.
  • the gesture determining module 220 may be configured to check whether the input signal corresponds to a predefined or predetermined relative gesture and optionally use gaze input signals to interpret the input signal.
  • the control module 200 may comprise a gesture storage unit (not shown) storing a library or list of predefined gestures, each predefined gesture corresponding to a specific input signal.
  • the gesture determining module 220 is adapted to interpret the received signals and provide, based on the interpreted signals, gesture based control commands, for example, a tap command to activate an object, a swipe command or a slide command.
  • a gaze data analyzing module 240 is configured to determine a gaze point area on the information presentation area 201 including the user's gaze point based on at least the gaze data signals from the gaze tracking module 235.
  • the information presentation area 201 may be a display of any type of known computer screen or monitor, as well as combinations of two or more separate displays, which will depend on the specific device or system in which the control module is implemented in.
  • the display 201 may constitute a regular computer screen, a stereoscopic screen, a heads-up display (HUD) in a vehicle, or at least one head-mounted display (HMD).
  • a processing module 250 may be configured to execute at least one user action manipulating a view presented on the information presentation area 201 based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • the user is able to control a device or system at least partly based on an eye-tracking signal which described the user's point of regard x, y on the information presentation area or display 201 and based on user generated gestures, i.e. a movement of at least one body part of the user can be detected, generating gesture based control commands via user input means 205 such as a touchpad.
  • the control module 260 is configured to acquire gesture based control commands from an input module 232'.
  • the input module 232' may comprise a gesture determining module and a data acquisition module as described above with reference to FIG. 13a.
  • a gaze data analyzing module 240 is configured to determine a gaze point area on the information presentation area 201 including the user's gaze point based on at least the gaze data signals received from the gaze tracking module 235.
  • the information presentation area 201 may be a display of any type of known computer screen or monitor, as well as combinations of two or more separate displays, which will depend on the specific device or system in which the control module is implemented in.
  • the display 201 may constitute a regular computer screen, a stereoscopic screen, a heads-up display (HUD) in a vehicle, or at least one head-mounted display (HMD).
  • a processing module 250 may be configured to execute at least one user action manipulating a view presented on the information presentation area 201 based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • the user is able to control a device or system at least partly based on an eye-tracking signal which described the user's point of regard x, y on the information presentation area or display 201 and based on user generated gestures, i.e. a movement of at least one body part of the user can be detected, generating gesture based control commands via user input means 205 such as a touchpad.
  • the input module 232" is distributed such that the data acquisition module 210 is provided outside the control module 280 and the gesture determining module 220 is provided in the control module 280.
  • a gaze data analyzing module 240 is configured to determine a gaze point area on the information presentation area 201 including the user's gaze point based on at least the gaze data signals received from the gaze tracking module 235.
  • the information presentation area 201 may be a display of any type of known computer screen or monitor, as well as combinations of two or more separate displays, which will depend on the specific device or system in which the control module is implemented in.
  • the display 201 may constitute a regular computer screen, a stereoscopic screen, a heads-up display (HUD) in a vehicle, or at least one head-mounted display (HMD).
  • a processing module 250 may be configured to execute at least one user action manipulating a view presented on the information presentation area 201 based on the determined gaze point area and at least one user generated gesture based control command, wherein the user action is executed with the determined gaze point area as a starting point.
  • the user is able to control a device or system at least partly based on an eye-tracking signal which described the user's point of regard x, y on the information presentation area or display 201 and based on user generated gestures, i.e. a movement of at least one body part of the user can be detected, generating gesture based control commands via user input means 205 such as a touchpad.
  • the wireless transmit/receive unit (WTRU) 300 is associated with an information presentation area 301 and further comprises input means 305, including e.g.
  • the WTRU further comprises a control module 200, 260 or 280 as described above with reference to FIGS. 13a, 13b and 13c.
  • the user is able to control the WTRU at least partly based on an eye- tracking signal which describes the user's point of regard x, y on the information presentation area or display 301 and based on user generated gestures, i.e. a movement of at least one body part of the user can be detected, generating gesture based control commands via user input means 305 such as a touchpad. All user actions described in the context of this application may also be executed with this embodiment of the present invention.
  • the computer device or handheld portable device 400 may, for example, be any one from the group of a personal computer, computer workstation, mainframe computer, a processor or device in a vehicle, or a handheld device such as a cell phone, smartphone or similar device, portable music player (such as e.g. an iPod), laptop computers, computer games, electronic books, an iPAD or similar device, a Tablet, a Phoblet/Phablet.
  • a personal computer computer workstation, mainframe computer, a processor or device in a vehicle, or a handheld device such as a cell phone, smartphone or similar device, portable music player (such as e.g. an iPod), laptop computers, computer games, electronic books, an iPAD or similar device, a Tablet, a Phoblet/Phablet.
  • the computer device or handheld device 400a is connectable to an information
  • the computer device or handheld device 400b includes an information presentation area 401b, as shown in FIG. 15b, such as a regular computer screen, a stereoscopic screen, a heads-up display (HUD), or at least one head-mounted display (HMD).
  • the computer device or handheld device 400a, 400b comprises input means 405 adapted to detect user generated gestures and a gaze tracking module 435 adapted to detect gaze data of a viewer of the information presentation area 401.
  • the computer device or handheld device 400a, 400b comprises a control module 200, 260, or 280 as described above with reference to FIG. 13a, 13b or 13c.
  • the user is able to control the computer device or handheld device 400a, 400b at least partly based on an eye-tracking signal which described the user's point of regard x, y on the information presentation area or display 401 and based on user generated gestures, i.e. a movement of at least one body part of the user can be detected, generating gesture based control commands via user input means 405 such as a touchpad. All user actions described in the context of this application may also be executed with this embodiment of the present invention.
  • FIG. 16-19 example embodiments of methods according to the present invention will be described.
  • the method embodiments described in connection with FIGS. 16-19 are implemented in an environment where certain steps are performed in a device, e.g. a WTRU described above with reference to FIG. 14, or a computer device or handheld device described above with reference to FIG. 15a or 15b and certain steps are performed in a control module, e.g. a control module as described above with reference to FIGS. 13a, 13b and 13c.
  • a control module e.g. a control module as described above with reference to FIGS. 13a, 13b and 13c.
  • the methods described herein can also be implemented in other devices, e.g. a WTRU described above with reference to FIG. 14, or a computer device or handheld device described above with reference to FIG. 15a or 15b and certain steps are performed in a control module, e.g. a control module as described above with reference to FIGS. 13a, 13b and 13c.
  • step S510 the user touches a touch sensitive area on the device (e.g. input means as described above) with one or more fingers of each hand.
  • a touch sensitive area on the device e.g. input means as described above
  • the gesture data i.e. the user input
  • step S530 it is checked whether the input signal corresponds to a predefined or predetermined relative gesture. If not, the procedure returns back to step S500.
  • a gesture based control command is generated at step S570.
  • the user looks at a screen or an information presentation area and at step S550 the user's gaze is detected at the information presentation area.
  • the step S540 is not a part of the method according to embodiments of the present invention.
  • a gaze point area including a user's point of gaze on the screen or information presentation area.
  • an action corresponding to the relative gesture at the user's point of gaze is performed based on the gesture based control command and the determined gaze point at the information presentation area.
  • step S590 the user makes a gesture with one or more fingers and/or at least one hand in front of the information presentation area (which gesture is interpreted by input means as described above).
  • the step S590 is not a part of the method according to embodiments of the present invention. There are a large number of conceivable gestures that the user can use to control actions of the device, and a non-exhaustive number of such gestures have been described above.
  • the gesture data i.e. the user input, is translated into an input signal.
  • step S530 it is checked whether the input signal corresponds to a predefined or predetermined relative gesture. If not, the procedure returns back to step S500.
  • a gesture based control command is generated at step S570.
  • the user looks at a screen or an information presentation area and at step S550 the user's gaze is detected at the information presentation area.
  • the step S540 is not a part of the method according to embodiments of the present invention.
  • a gaze point area including a user's point of gaze on the screen or information presentation area.
  • an action corresponding to the relative gesture at the user's point of gaze is performed based on the gesture based control command and the determined gaze point at the information presentation area.
  • step S592 the user generates input by touching touchpad or predefined area of touch-screen.
  • the step S592 is not a part of the method according to embodiments of the present invention. There are a large number of conceivable gestures that the user can use to control actions of the device, and a non- exhaustive number of such gestures have been described above.
  • the gesture data i.e. the user input
  • the gesture data is translated into an input signal.
  • a gesture based control command is generated at step S570.
  • the user looks at a screen or an information presentation area and at step S550 the user's gaze is detected at the information presentation area.
  • the step S540 is not a part of the method according to embodiments of the present invention.
  • a gaze point area including a user's point of gaze on the screen or information presentation area is determined.
  • an action corresponding to the relative gesture at the user's point of gaze is performed based on the gesture based control command and the determined gaze point at the information presentation area.
  • step S594 the user generates input by making a gesture with one or more of his or hers fingers and/or at least one hand.
  • the step S594 is not a part of the method according to embodiments of the present invention. There are a large number of conceivable gestures that the user can use to control actions of the device, and a non-exhaustive number of such gestures have been described above.
  • the gesture data i.e. the user input
  • step S530 it is checked whether the input signal corresponds to a predefined or predetermined relative gesture. If not, the procedure returns back to step S500.
  • a gesture based control command is generated at step S570.
  • the user looks at a screen or an information presentation area and at step S550 the user's gaze is detected at the information presentation area.
  • the step S540 is not a part of the method according to embodiments of the present invention.
  • a gaze point area including a user's point of gaze on the screen or information presentation area is determined.
  • an action corresponding to the relative gesture at the user's point of gaze is performed based on the gesture based control command and the determined gaze point at the information presentation area.
  • a gaze tracking module (not shown) and a user input means 900 are implemented in a vehicle (not shown).
  • the information presentation area (not shown) may be a heads-up display or an infotainment screen.
  • the input means 900 may be one or two separate touch pads on the backside (for use with the index finger/s) or on the front side (for use with the thumb/s) of the steering wheel 910 of the vehicle.
  • a control module 950 is arranged in a processing unit configured to be inserted into a vehicle or a central processing unit of the vehicle.
  • the control module is a control module as described with reference to FIGS. 13a-13c.
  • a gaze tracking module (not shown) and an information presentation area (not shown) are implemented in a wearable head mounted display 1000 that may be designed to look as a pair of glasses.
  • the user input means 1010 may include a gyro and be adapted to be worn by the user 1020 on a wrist, hand or at least one finger.
  • the input means 1010 may be a ring with a wireless connection to the glasses and a gyro that detects small movements of the finger where the ring is worn.
  • the detected movements representing gesture data may then wirelessly be communicated to the glasses where gaze is detected and gesture based control commands based on the gesture data from the input means is used to identify and execute user action.
  • a control module as described with reference to FIG. 13a-13c is used with this implementation.
  • the user 1120 is able to control a computer device 1100 at least partly based on an eye-tracking signal which describes the user's point of regard x, y on an information presentation area 1140 and based on user generated gestures, i.e. a movement of at least one body part of the user can be detected, generating gesture based control commands via user input means 1150.
  • the user 1120 can generate the gesture based control commands by performing gestures above or relative the keyboard of the computer device 1100.
  • the input means 1140 detects the gestures, for example, using an optical measurement technique or capacitive measurement technique.
  • a control module as described with reference to FIG.
  • the computer device 1100 may, for example, be any one from the group of a personal computer, computer workstation, mainframe computer, or a handheld device such as a cell phone, portable music player (such as e.g. an iPod), laptop computers, computer games, electronic books and similar other devices.
  • the present invention may also be implemented in an "intelligent environment" where, for example, objects presented on multiple displays can be selected and activated.
  • a gaze tracker unit (not shown) is included in the computer device 1100, or is associated with the information presentation area 1140.
  • a suitable gaze tracker is described in the U.S. Pat. No.
  • a system and method for adapting the interface of a portable device based on information derived from a user's gaze comprises a portable device containing, or operatively linked to, an eye tracking device, where the eye tracking device is adapted to determine a user's gaze relative to the portable device.
  • the portable device contains a display, and a module for displaying information on that display. This module is typically part of an operating system.
  • Popular operating systems include the Google AndroidTM and Apple iOSTM operating systems.
  • a module in the portable device operatively connected with the eye tracking device compares a user's gaze information with information displayed on the portable device display at the time of the user's gaze.
  • the user's gaze or eye information may be used to identify or otherwise log the user in to a portable device.
  • a gaze pattern may be used to identify a user
  • iris identification may be used
  • facial identification using facial features may be used, as would be readily understood by a person of skill in the art.
  • a user may be identified through non-gaze means, such as traditional pattern or password based login procedures.
  • information displayed may be modified based on that identity. This modified information may be combined with gaze information of the user, to provide for an improved user experience. For example, if a user is recognized by the device, when the device is in a limited mode such as a locked mode, more information is displayed than if a user is not recognized by the device.
  • a user may be identified and the history of that user's usage of the portable device logged. This identification can be used in many contexts. For example, the identity of the user may be provided to applications running on the protable device, the applications may then modify their behavior or store the usage information for the particular user. As a further example, when the user gazes at a contact in a phone application, that is either linked to the user's profile or frequently contacted, the phone application may instantly place a video or audio call and display the call on the display.
  • User identification may be utilized to identify a user in the context of a website or other application.
  • a shopping application or website may determine a user is the registered owner of an account based on the user's eyes (for example iris identification) or gaze pattern.
  • the application or website may also sign a user out, once the portable device determines the user is no longer gazing at the application or website within a predefined period of time, or that the original user to open the application or website is no longer present in front of the device.
  • a system and method for adapting the interface of a portable device based on information derived from a user's gaze comprises a portable device containing, or operatively linked to, an eye tracking device, where the eye tracking device is adapted to determine a user's gaze relative to the portable device.
  • the portable device contains a display, and a module for displaying information on that display. This module is typically part of an operating system.
  • Popular operating systems include the Google AndroidTM and Apple iOSTM operating systems.
  • a module in the portable device operatively connected with the eye tracking device compares a user's gaze information with information displayed on the portable device display at the time of the user's gaze.
  • This data is collected over time and may be analyzed by the portable device according to the present invention, in order to alter further information shown on the display. [0199]
  • the portable device, or software executed on the portable device may define that certain information has been "seen”. Information that has been seen has not necessarily been read or understood by a user, but has merely been noticed by the user. Any seen information could be removed, or altered in how it is displayed until a user determines to read the information in more detail.
  • the portable device may determine that the user is gazing at an email application and thus may show unread emails.
  • the portable device may determine that a user typically gazes at unread emails before gazing at unread messages, and thus may make unread emails appear before unread messages.
  • an application running on the portable device may display information about the weather.
  • the application such as an icon
  • the item animates. For example, if the weather is warm a picture of a sun may move.
  • an application running on the portable device may display an avatar (an image of a person).
  • an avatar an image of a person
  • it may instead display a view from an image sensor on the portable device (for example, an image of the user of the portable device).
  • the avatar may animate so as to react differently to the location of the user's gaze on the avatar.
  • an application running on the portable device may display images, or reduced resolution versions of images. These images may have been captured by an image sensor incorporated in the portable device, or collected by the device in other ways, such as downloaded from the internet.
  • the application may sort display of these images according to the amount of times they have been viewed, the duration they have been viewed for, or any other metric definable by a user's gaze.
  • a user may use an application containing summaries of further information, for example an application showing thumbnails (reduced resolution versions) of images, upon gazing at a thumbnail the full resolution image of that thumbnail, or related images, may be displayed.
  • an application showing thumbnails (reduced resolution versions) of images, upon gazing at a thumbnail the full resolution image of that thumbnail, or related images, may be displayed.
  • the portable device may have an image displayed, such as a background image.
  • the image may dynamically modify in line with the user's gaze. For example, consider an image of a night sky containing a plurality of stars. As the user gazes across the image, the starts located at the user's gaze may highlight by changing size, shape, color etc.
  • information may be maintained on a display for as long as a users' gaze remains fixated on or near the information.
  • information may be relayed to a remote person or location regarding the user's gaze information.
  • an application allowing text communication between two or more parties such as an SMS application, may display on a remote device that a message is being read or otherwise gazed at by a user of a local device.
  • a system and method for adapting the interface of a portable device based on information derived from a user's gaze comprises a portable device containing, or operatively linked to, an eye tracking device, where the eye tracking device is adapted to determine a user's gaze relative to the portable device.
  • the portable device contains a display, and a module for displaying information on that display. This module is typically part of an operating system.
  • Popular operating systems include the Google AndroidTM and Apple iOSTM operating systems.
  • a module in the portable device operatively connected with the eye tracking device compares a user's gaze information with information displayed on the portable device display at the time of the user's gaze.
  • This gaze information may be linked to an item displayed on the display at the location of the user's gaze and stored. This stored information may then be used by an application stored on the portable device. In this way, the context of the information gazed at by a user may be used by multiple applications on the portable device. For example, a user may gaze at a location on a map displayed on the display, then when the user accesses another application, such as a web browser, that location may be used to display customized results.
  • another application such as a web browser
  • a user may gaze at information on the display and that information may be temporarily stored by the portable device and provided to applications on the portable device. This is best illustrated by the case where a user views a timetable on their device, such as a bus timetable. If the user then loads an application to transmit information about the bus times, such as a messaging application, according to the present invention the time of the last viewed bus, or the time of the bus that was most often viewed, can be automatically provided to the messaging application.
  • a timetable such as a bus timetable.
  • This same invention can be applied to many use cases, for example images can be inserted into messages or emails based on the user's gaze history or a user looking at recipes, shopping lists etc can have their viewed at information provided to shopping applications or web browsers, so as to expedite the process of searching of the items in the shopping list, recipe etc.
  • a user may have recently installed a new application on their portable device. Upon first use, a certain number of uses, until disabled by the user, or always, the application may provide visual information to the user dependent upon their gaze location. For example, if the application was a mail application, when the user gazed at an icon that provides the ability to write new mail messages, the icon may visually highlight via a change in color, position, size etc.
  • a user may be utilizing an application that offers extended functionality.
  • a map application may show nearby restaurants if a menu is enabled.
  • gaze information an indication of this extended functionality may be offered to the user.
  • the nearby restaurants information may normally be identified by "swiping" (moving ones finger across the display) from the side of the display.
  • gaze information when the user looks at his location on the map, the extended functionality may slightly appear from the side of the display, demonstrating to the user that more functionality may be achieved by swiping from that side of the display.
  • An item is displayed on a display on a portable device.
  • An eye tracking device determines a user's gaze relative to the display.
  • the portable device determines that a user's gaze is, or was, located on or near the item. Based on the duration of the gaze on the item, the portable device can determine that the user is: o Glancing at the item, in which case the portable device will take no action. o Interested in the item, in which case the portable device can respond by
  • the portable device can respond by taking a second action.
  • the item is an image of a button, the first action is visually highlighting the button, and the second action is providing a visual hint as to the effect of activating the button.
  • the item is a reduced resolution version of an image (aka a thumbnail), the first action is visually highlighting the thumbnail, and the second action is enlarging the thumbnail.
  • the determination of whether the user is interested or very interested can be based on total time the user gazes at the item, where once the user has gazed at the icon for a predetermined length of time, the device determines the user is interested or very interested. Alternatively, the determination could similarly be based on frequency of times the user gazes at the item.
  • the determination could be based on the history of the user's usage of the portable device.
  • a system and method for adapting the interface of a portable device based on information derived from a user's gaze comprises a portable device containing, or operatively linked to, an eye tracking device, where the eye tracking device is adapted to determine a user's gaze relative to the portable device.
  • the portable device contains a display, and a module for displaying information on that display. This module is typically part of an operating system.
  • Popular operating systems include the Google AndroidTM and Apple iOSTM operating systems.
  • a module in the portable device operatively connected with the eye tracking device compares a user's gaze information with information displayed on the portable device display at the time of the user's gaze.
  • This gaze information may be linked to an item displayed on the display at the location of the user's gaze and stored. This stored information may then be used by an application stored on the portable device. This information may be used, for example, to determine that user is facing the display and therefore power saving features such as dimming of the display may not occur. Further, for example, sound emitted by the device either universally or for a specifics application or specific period, may be muted or otherwise reduced in volume while a user is facing or looking at the display.
  • the presence of a user, or the gaze of a user may be used to modify the contents of the display, or behavior of the portable device.
  • an application may be provided on a portable device allowing a user to set a timer, in other words a countdown from a numerical value to zero. When the timer reaches zero, typically an alarm will sound to notify the user that the timer has reached zero.
  • the alarm may be silenced when the user gazes at the portable device.
  • the mere presence of the user's face near the portable device may cause the alarm to silence.
  • Determination of the user's presence may be initiated by the portable device indicating it is in an active mode, which could be triggered by an accelerometer, specific application running on the device etc. Once the user's presence has been established, functionality on the device may be altered accordingly.
  • the display may increase or decrease in brightness
  • audio may increase or decrease in volume
  • text may be obscured or revealed
  • the order of items on the display may be changed etc.
  • a portable device is receiving a phone call.
  • the device may emit a ring tone, without powering on the display until a user's presence is detected.
  • the device could then determine that either the user has been present for a predetermined period of time, or the user has gazed at certain information (such as the caller identification), and decrease or mute the volume of the ring tone.
  • Presence based information could also be modified based on the identity of the user, if the user has been identified. For example, the identity of the user could determine how much information is displayed. Take a text message for example, an unidentified user may not be able to read the contents of the message, while an identified user may.
  • a system and method for adapting the interface of a portable device based on information derived from a user's gaze comprises a portable device containing, or operatively linked to, an eye tracking device, where the eye tracking device is adapted to determine a user's gaze relative to the portable device.
  • the portable device contains a display, and a module for displaying information on that display. This module is typically part of an operating system.
  • Popular operating systems include the Google AndroidTM and Apple iOSTM operating systems.
  • a module in the portable device operatively connected with the eye tracking device compares a user's gaze information with information displayed on the portable device display at the time of the user's gaze.
  • This gaze information may be linked to an item displayed on the display at the location of the user's gaze and stored. This stored information may then be used by an application stored on the portable device. For example, the information may be used to highlight, or otherwise visually mark items on a display to draw a user's attention. For example, it is typical in the operating system of a portable device to allow for many applications to be loaded and run on the device. It can be difficult for a user to understand which applications may be interacted with, or within an application, which sections, icons etc may be interacted with.
  • elements on the display may demonstrate that they may be interacted with, or even how they may be interacted with, when a user gazes at the element. This demonstration includes, but is not limited to, changing of color, size, displaying new colors, animations, and changing images.
  • a portable device displays a visual indication based on gaze such as a highlight, animation etc. as has been previously described, and the gaze signal is momentarily lost, it is beneficial to have a soft transition so as to minimize the effect of the lost signal on the user.
  • a visual indication being a highlighting of an icon on the display
  • the visual indication is a hard on/off style transition
  • the highlight will immediately cease. This creates an abrupt experience for the user. If the highlight appears in a transitional manner, as shown in Fig. 25 where the amount of highlight increases gradually, then when the gaze signal is momentarily lost, the highlight beings to gradually decrease. If the gaze signal is resumed, the highlight may gradually increase again. This creates an easier, more natural experience for the user.
  • a user's gaze information is used to determine an item on a display in which the user wishes to interact. This determination could be based on gaze duration, frequency of gaze, gaze history or any other metric described in this application.
  • interaction with that item can continue even if the user gazes away from the item.
  • Interaction could be touch based, gesture based, voice based or any other form of interaction conceivable.
  • the interaction with the item ceases when the interaction itself ceases.
  • a user may gaze at an icon on a display which controls a dimmable light in a room.
  • the level of brightness of the light may be adjusted by sliding a finger across the display.
  • the user gazes at the icon then places their finger on the display and moves their finger to control the brightness. Regardless of the location of the user's gaze during the sliding gesture, operation of the light switch will not be interrupted. Once the sliding gesture is complete and the user's finger is removed from the display, normal operation of the device resumes.
  • the subject of a user's gaze may be combined with a non-gaze input to provide functionality on a portable device. For example a user may gaze at an item on a display and speak a specific word, the portable device may then execute a function based on the spoken word and the item gazed at on the display. To demonstrate, a user may gaze at a list of contacts on a display and linger their gaze on a specific contact. The user may then say "call" for example, and the portable device will call the phone number associated with the contact the user's gaze is lingering on.
  • the user's gaze information may be collected to improve the accuracy of the non-gaze input. For example, frequently gazed at information on a display can be more readily used by non-gaze input enabled applications. This may be used by predictive text algorithms, as would be readily understood by a person skilled in the art. Frequently gazed at words, phrases or letters may be collected and used to predict text that is being typed by a user.
  • the context of the non-gaze input may be modified by the location of a user's gaze.
  • the user may gaze at a text field in an internet search engine, and when the user speaks a word, the search engine may search for results relevant to that world. Whereas if the user was to speak that same word while the user was gazing elsewhere than the text field, then a different function or possibly no function would be executed.
  • the information displayed on a display may be in the form of an avatar.
  • An avatar is a graphical representation of a person, animal, or any other such being.
  • a portable device may provide a more personable experience to a user.
  • a user's gaze may be determined and once recognized by the portable device that a user's gaze is directed towards or near a displayed avatar, the avatar may respond in some fashion.
  • the response of the avatar includes, but is not limited to:
  • a user may interact with a portable device using their voice only when they intend. Without gaze being determined to be towards or near the avatar, the portable device will not analyze the user's voice for spoken commands. However, certain commands may still be spoken towards the portable device without the user's gaze direction being detected towards the display. For example, a spoken command such as "where are you?" may still cause the portable device to respond.
  • improved input to a portable device using a touch based keyboard is proposed.
  • This embodiment allows a user to conveniently enter text into a portable device without requiring stretching of fingers across the portable device to select a text input field.
  • the information displayed on the display comprises a text input field, when a user's gaze is determined by the portable device to be direct to or near the text input field, a keyboard is displayed on the screen.
  • a user may contact the device with a finger to enable the display of the keyboard, this may be a touch or swiping motion by the user. This contact may occur on the display, on a physical button or outside of the portable device.
  • the user may enter text into the text input field in a manner as would be recognized by anyone of skill in the art, in fact by anyone who has ever used a portable device with a touch interface.
  • the information displayed on the display is icons or other such information that may be selected by a user.
  • predetermined area of the display such as the top, icons may be enlarged. By then gazing at an enlarged icon it may be selected by pressing on the portable device in some fashion.
  • the enlargement of icons may be performed upon a touch input such as placing a finger on the screen in a predetermined area, or on a fingerprint sensor of the like.
  • gaze may be used to determine which icon is to be selected and when the user removes their finger from the screen or fingerprint sensor, or performs a deliberate action such as swiping , clicking or the like, an application associated with the selected icon may be opened.
  • enlargement may be optional and fine adjusting of the gaze direction can be performed through contact with the display or a fingerprint sensor.
  • a user's gaze direction may be shown on the display, or an icon highlighted, and by moving a finger on the screen or sensor this direction or highlight may be moved proportionally.
  • interaction such as a click with the portable device may be achieved by touching any location on the portable device and gazing at the subject of the action on the display.
  • This touch may be in a predetermined area on the display, or be performed in a certain manner, for example 2 touches in quick succession.
  • icons showing currently running processes and applications may be displayed upon a user's gaze being directed towards a predetermined area of the screen. The user may then gaze towards an icon representing a currently running application the user desires to activate, upon dwelling their gaze upon the icon or contacting the portable device in a predetermined manner, the selected application may be made active on the portable device.
  • information may be displayed on the display during a "locked" or limited functionality phase of the portable device.
  • a phase is used to show notifications such as missed calls, messages, reminders and the like.
  • Highlighting may include brightening of the notification, otherwise graphically separating the notification from other items on the display, or displaying further information regarding the notification.
  • Selecting a notification may be performed by touching the display or a fingerprint sensor, or by dwelling gaze on the notification for a predetermined period of time, for example 0.1 to 5 seconds.
  • a notification may expand and/or separate, allowing for easier gaze determination towards each notification.
  • more information regarding that notification may be displayed. For example if the notification is a text message, the name of the person from which the text message may be initially displayed, and then the entire text message displayed in an expanded view.
  • the fingerprint sensor may recognize a user's fingerprint and unlock the portable device and open an application associated with the notification.
  • a notification is gazed at for a predetermined period of time, for example 1 to 5 seconds. The portable device may cease to display the notification .
  • a method for lowering an audible sound from a portable device for example, music, video sound, ring tones, alerts, warnings, etc.
  • a portable device for example, music, video sound, ring tones, alerts, warnings, etc.
  • the portable device emits a sound.
  • An eye determination portion of the portable device i.e., an eye tracking device or other image sensor (i.e., camera or other image/video capture device) determines that a user is gazing towards the portable device.
  • the volume decrease described in step 3 may be a gradual decrease, or an instantaneous decrease - otherwise known "mute.”
  • the decrease may be total (volume to zero), or may be to a predetermined lower level (either an absolute level or a percentage level relative to the original volume).
  • a difference audio content e.g., a notification of gaze recognition and/or dismissal of the original audio content
  • a choice of the preferred method of audio change, along with its associated parameters, may be determined by a user and stored within the portable device, for use during step 2.
  • the determination of a user's gaze in step 2 may be based on a determination that a user has gazed anywhere within the vicinity of the portable device (for example, within five centimeters of the device), or upon a specific, predetermined area of the portable device (for example, the screen, keyboard, and/or a particular side of the portable device).
  • the determination may not be based upon gaze at all, but rather upon a
  • an image sensor in the portable device i.e., camera or other image/video capture device
  • Examples of suitable sounds that may be altered by this embodiment are ringtones of a portable device as well as an alarm emitted by a portable device.
  • Fig. 2410 it is determined whether the portable device is emitting audio content. If so, method 2400 recognizes that such audio content may change should a particular gaze event occur (i.e., particular eye information such as gaze direction, eye presence, and/or eye position).
  • a gaze event i.e., particular eye information such as gaze direction, eye presence, and/or eye position.
  • a gaze event it is determined whether a gaze event has occurred. As discussed above, this could result from the mere detection of an eye of the user, or of a particular gaze direction of the user.
  • the audio content is changed based on the gaze event.
  • a gesture such as a swipe may be performed on a touch-sensitive surface of the portable device.
  • a user may perceive the sensation of "pulling in” something located off screen. For example the user may gaze above the portable device, while simultaneously pulling the phone in a downwards motion, causing a menu to appear on the display from the top of the portable device. This gives the user a feeling of looking at an invisible menu that exists above the portable device, and then pulling that menu in by moving the device downwards.
  • a method for activating a portable device and enabling gaze tracking comprises the following steps:
  • Step 2 may be achieved by shaking or touching the portable device in a predetermined manner.
  • the present embodiment may function in the following manner.
  • a portable device is programmed to switch to a battery saving inactive mode after a certain period of time where the device is not used, for example the screen may be switched off.
  • this inactive mode any hardware and software used for gaze tracking is typically disabled.
  • the portable device Upon shaking the portable device, the portable device is switched to active mode and the gaze tracking hardware and software is enabled.
  • a portable device receives a telephone call.
  • the portable device notifies a user of the call through a visual representation.
  • Handsfree mode is intended to refer to a commonly known method of handling telephone calls on portable devices, whereby a microphone and speaker in the portable device are operated such that a user can participate in a telephone call without physically contacting the portable device.
  • a handsfree mode may also be used with external devices such as a headset, external speakers and/or external microphones.
  • the visual representation in step 2 is preferably an icon or image representing answering a telephone call, for example a green image of a telephone may be used.
  • the device may answer the call and enter handsfree mode. Alternatively, if the device determines the user is performing a "shaking" motion with their head, the device may refuse the call.
  • Embodiments Involving a Portable Device as an Information Device
  • a portable device may be used to display information pertinent to the surrounds of the device, based on a gaze direction of a user.
  • a portable device determines a user's gaze direction relative to the portable device.
  • the portable device determines positional information of the portable device.
  • the portable device receives information relevant to the subject of the user's gaze direction.
  • the portable device displays such information.
  • Step 2 of this method requires that the portable device determine positional information, this may be utilizing a global positioning system receiver (GPS), accelerometer, gyroscope or similar or a combination thereof.
  • GPS global positioning system receiver
  • Step 3 of this method requires that the portable device receive information relevant to the subject of the user's gaze, optionally this information may already be stored in the portable device and instead of receiving the information, the portable device retrieves the information from memory .
  • the information may be directed to the portable device through multiple methods, including:
  • the portable device may utilize a GPS system to determine its location, and download suitable information.
  • the portable device may receive information from a nearby transmitter.
  • a user may use a portable device in an environment such as a museum, the portable device may have stored information relevant to exhibits in the museum along with positional information of the exhibits. As a user moves around the museum, the portable device determines that the user's gaze is directed away from the portable device. Using the positional information of the device, the portable device can determine exhibits located close to it, as well as which exhibit the user is gazing towards. Thus the portable device can display information relative to that exhibit. [0307] Embodiments Involving Page Flipping
  • a method for moving from one page to another in a book or similar displayed on a portable device This embodiment functions with only the determination of eye position of the user.
  • the method functions with the following steps:
  • Determination of the user's eye position may be performed using any image sensor of the device.
  • the position may be compared to a "normal" position whereby a user's eye or eyes is in a substantially horizontal position.
  • the orientation of the user's eyes is used to determination in which direction to turn the pages of the multipage document. For example, if the orientation is such that the user has tilted their head to the right, the next page will be displayed and vice-versa.
  • the device may skip displayed pages, for example if the user continues to tilt their head, multiple pages may be passed over until the user straightens their head.
  • the visual indicator described in step 3 is preferably an animation of a page turning, but may be any form of image, video, sound or the like.
  • a display is changed on a portable device according to the distance of the device from a user.
  • the embodiment contains the following steps:
  • the movement of the device may be forward-backwards or side to side.
  • the device may be held in front of a user and moved backwards until the desired menu of items is reached.
  • the embodiment may function in the opposite manner whereby when a user moves his/her head relative to the device, the displayed information changes.
  • the device determines a user's eye position and tracks any relative changes in the eye position, if it determines that the eye position indicates the user is closer or further away from the device, information on the display is changed. For example if the device is displaying a map, and it determines that the user is moving his/her head towards the map, the device may display a zoomed in/enlarged view of a portion of the map.
  • the GPS of a portable device may only be activated when the portable device determine that a user is present in front of the device .
  • the determination of a user's presence is performed by analyzing an image captured by an image sensor on the device, whereby the analysis looks for the presence of a user's eye or eyes.
  • the device may disable any image sensors or associated hardware or software. For example, the device may determine by the proximity of a user's eyes that the user is too close to the display to accurately perform eye tracking or eye position determination. The device may also determine, through information obtained by capturing images with the image sensor, that the device is upside down, on a surface, in a pocket etc.
  • further sensors found in the portable device may be used to complement information obtained by a gaze tracking or eye position determination system.
  • information from the accelerometer may be used to determine if the device has moved and this information can be compared to information from the eye tracking or eye position determination system to decide if the device has moved, or the user's head has moved.
  • a user's attention level to a particular display or other stimuli application may be classified as falling within one or more of a particular number of categories.
  • three possible attention levels may be defined as “no-attention," "facing,” or “looking-at.”
  • No-attention may occur when a gaze detection system determines that no user is present (i.e., there are no eyes to track).
  • “Facing” may occur when a gaze detection system determines that a user is at least present and their gaze is at least facing the device (i.e., eyes are visible to the gaze detection system).
  • Looking-at may occur when a gaze detection system determines that a gaze is more particular detected in some predefined region (or the entirety) of a display.
  • a given hardware component or displayed graphical component may also be classified as falling within one or more of a particular number of categories regarding the user's attention thereto.
  • six possible attention levels may be defined as “unobserved,” “faced,” “glanced-at,” “viewed,” “seen,” and “interesting-to-user.”
  • "Unobserved” may be established as a component status when a gaze detection system determines that the component has not yet been looked at by a user (i.e., the eyes of the user have not been detected).
  • “Faced” may be established as a component status when a gaze detection system determines that the component has not yet been looked at by a user, but the user is generally facing the component (i.e., eyes detected, but not gaze on the component).
  • “Glanced-at” may be established as a component status when a gaze detection system determines that the component has been gazed at by the user for at least some minimum threshold of time, but not longer than some maximum threshold of time.
  • “Viewed” may be established as a component status when a gaze detection system determines that the component has been gazed at by the user for at least some minimum threshold of time (i.e., longer than "glanced-at”).
  • “Seen” may be a highly contextual status which is established as a component status when a gaze detection system determines that the user's gaze has interacted with the component in at least a certain manner, which may vary depending on the content of the component). Merely by way of example, various factors, depending on the content of the component, may be examined such as the amount of time the user's gaze has remained or revisited the component, the pattern of the gaze (i.e., direction it moves to, and around, the component), etc. An elevated version of the "seen” status may be "interesting-to-user” and may rely on similar factors, but require a greater magnitude of agreement therewith.
  • Taking into account the above user attention levels and component statuses may allow for more useful interactions and/or smarter predictions about user intentions and/or needs.
  • the user attention levels and component statuses may thus supplement any other gaze detection efforts and algorithms discussed herein or elsewhere. This may especially be the case in mobile-device applications, where traditional input systems are either limited or not present (i.e., full size keyboards, and mice). But such methods discussed herein can also at least supplement tradition input systems regardless.
  • face identification via gaze detection systems may also supplement or replace other identification verification systems at the same time.
  • one possible interaction with a mobile or other device which may be improved is by input prediction. If a user is regularly observed gazing at photos of certain objects, the name of that object may be more likely to appear in a typing interface where words are suggested to the user as they type. For example, a touch keyboard application or a search prompt may suggest the name of the object as a possible input. Context of the current situation (e.g., other nearby words and/or the particular application being used) may be used by the algorithm to further predict precisely when such a suggested input would be necessary.
  • sorting of various items can occur based on previous gaze patterns. For example, a list of applications, documents, or media files may be sorted based on previous viewing patterns, with often looked at items being placed higher or to one end in a list or arrangement than less often looked at items.
  • gaze inputs may be used to supplement other input modes and/or other applications from which the gaze data is received to increase later input accuracy.
  • names of often gazed at items could be added to speech recognition input means, and or narrow search results quickly.
  • Preferences for operating system and/or application elements could also be based at least partially on gaze detection information.
  • recipients for transmitted information could also, over time, be associated with particular gaze patterns of a user, as well as particular content gazed at by the user.
  • transient graphical components may only be displayed for default time periods to a user before disappearing or otherwise being minimized.
  • email clients may display "toast" or other notifications which will remain for a set period of time on the display before being removed or minimized.
  • Embodiments herein may determine that a user is gazing upon such notifications, and extend, either for a predefined period of time, or indefinitely, display of the notification. Any time-dependent sleep or lock modes of an application/device may also be extended in this manner, such that sleep/lock modes are delayed so long as user gaze is detected.
  • facial recognition via a gaze-detection or other system may allow for a user's identity to be continually verified. So long as verification occurs, access to security- sensitive applications may continue, but end when facial recognition is no longer possible. This may allow for automatic login and/o shutdown of security sensitive applications when a user begins/stops observing the application.
  • a guest mode may be available for an application or device which provides a different interface for the owner or regular user of the device than that which may be provided to a guest, based on who the gaze detection system believes is using the device.
  • gaze requirements may be set for data which is transmitted between parties so that only certain intended recipients of the data can retrieve the information.
  • facial, iris, or some other facial identification may be required for a transmitted message to be opened or readable.
  • an application may inform a user graphically or otherwise of changes that have occurred since a component was previously viewed. This may allow a user to quickly understand what changes have occurred since a prior viewing.
  • gaze information may be transmitted to other parties to enhance communication there-between. For example, if two users are viewing a textual conversation on different devices, each person may be informed in some manner of what portion of the conversation is being observed by the other person.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Position Input By Displaying (AREA)

Abstract

L'invention concerne un procédé et un système d'aide à un utilisateur lors de l'interaction avec une interface utilisateur graphique, qui combinent une entrée basée sur le regard avec des commandes d'utilisateur basées sur un geste. Un utilisateur d'un système informatique sans un écran tactile classique peut interagir avec des interfaces utilisateur graphiques d'une manière similaire à un écran tactile à l'aide d'une combinaison d'entrées basées sur un regard et des commandes utilisateur basées sur un geste. Une solution pour une interaction de type écran tactile utilise une entrée de regard et une entrée basée sur un geste en tant que complément ou une alternative à des interactions tactiles avec un dispositif informatique ayant un écran tactile. Des interactions combinées basées sur le regard et le geste avec des interfaces utilisateur graphiques peuvent être utilisées pour obtenir un environnement analogue à un écran tactile dans des systèmes informatiques sans écran tactile classique ou dans des systèmes informatiques ayant un écran tactile dont la conception ergonomique est défavorable pour l'utilisateur ou un écran tactile agencé de telle sorte qu'il soit plus confortable pour l'utilisateur d'utiliser le geste et le regard pour l'interaction que l'écran tactile.
PCT/US2018/019447 2017-02-27 2018-02-23 Système d'interaction de regard WO2018156912A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/444,035 2017-02-27
US15/444,035 US20170235360A1 (en) 2012-01-04 2017-02-27 System for gaze interaction

Publications (1)

Publication Number Publication Date
WO2018156912A1 true WO2018156912A1 (fr) 2018-08-30

Family

ID=61599645

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2018/019447 WO2018156912A1 (fr) 2017-02-27 2018-02-23 Système d'interaction de regard

Country Status (1)

Country Link
WO (1) WO2018156912A1 (fr)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110286771A (zh) * 2019-06-28 2019-09-27 北京金山安全软件有限公司 交互方法、装置、智能机器人、电子设备及存储介质
CN111405497A (zh) * 2018-12-30 2020-07-10 云丁网络技术(北京)有限公司 一种短消息处理方法及装置
CN111796752A (zh) * 2020-05-15 2020-10-20 四川科华天府科技有限公司 一种基于pc的互动教学系统
US11281291B1 (en) 2020-10-20 2022-03-22 Rovi Guides, Inc. Methods and systems of extended reality environment interaction based on eye motions
US11320903B1 (en) 2020-10-20 2022-05-03 Rovi Guides, Inc. Methods and systems of extended reality environment interaction based on eye motions
US11392198B2 (en) 2020-10-20 2022-07-19 ROVl GUIDES, INC. Methods and systems of extended reality environment interaction based on eye motions
US11609629B2 (en) 2020-10-20 2023-03-21 Rovi Guides, Inc. Methods and systems of extended reality environment interaction based on eye motions
US11747896B2 (en) * 2020-10-20 2023-09-05 Rovi Guides, Inc. Methods and systems of extended reality environment interaction based on eye motions
CN110286771B (zh) * 2019-06-28 2024-06-07 北京金山安全软件有限公司 交互方法、装置、智能机器人、电子设备及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6204828B1 (en) 1998-03-31 2001-03-20 International Business Machines Corporation Integrated gaze/manual cursor positioning system
US20050243054A1 (en) 2003-08-25 2005-11-03 International Business Machines Corporation System and method for selecting and activating a target object using a combination of eye gaze and key presses
US7572008B2 (en) 2002-11-21 2009-08-11 Tobii Technology Ab Method and installation for detecting and following an eye and the gaze direction thereof
US20090273562A1 (en) * 2008-05-02 2009-11-05 International Business Machines Corporation Enhancing computer screen security using customized control of displayed content area
US8235529B1 (en) 2011-11-30 2012-08-07 Google Inc. Unlocking a screen using eye tracking information
EP2503479A1 (fr) * 2011-03-21 2012-09-26 Research In Motion Limited Procédé de connexion basé sur le sens du regard
US20140313307A1 (en) * 2013-04-18 2014-10-23 Samsung Electronics Co., Ltd. Electronic device and method for unlocking in the electronic device

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6204828B1 (en) 1998-03-31 2001-03-20 International Business Machines Corporation Integrated gaze/manual cursor positioning system
US7572008B2 (en) 2002-11-21 2009-08-11 Tobii Technology Ab Method and installation for detecting and following an eye and the gaze direction thereof
US20050243054A1 (en) 2003-08-25 2005-11-03 International Business Machines Corporation System and method for selecting and activating a target object using a combination of eye gaze and key presses
US20090273562A1 (en) * 2008-05-02 2009-11-05 International Business Machines Corporation Enhancing computer screen security using customized control of displayed content area
EP2503479A1 (fr) * 2011-03-21 2012-09-26 Research In Motion Limited Procédé de connexion basé sur le sens du regard
US8235529B1 (en) 2011-11-30 2012-08-07 Google Inc. Unlocking a screen using eye tracking information
US20140313307A1 (en) * 2013-04-18 2014-10-23 Samsung Electronics Co., Ltd. Electronic device and method for unlocking in the electronic device

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111405497A (zh) * 2018-12-30 2020-07-10 云丁网络技术(北京)有限公司 一种短消息处理方法及装置
CN111405497B (zh) * 2018-12-30 2021-12-24 云丁网络技术(北京)有限公司 一种短消息处理方法及装置
CN110286771A (zh) * 2019-06-28 2019-09-27 北京金山安全软件有限公司 交互方法、装置、智能机器人、电子设备及存储介质
CN110286771B (zh) * 2019-06-28 2024-06-07 北京金山安全软件有限公司 交互方法、装置、智能机器人、电子设备及存储介质
CN111796752A (zh) * 2020-05-15 2020-10-20 四川科华天府科技有限公司 一种基于pc的互动教学系统
US11281291B1 (en) 2020-10-20 2022-03-22 Rovi Guides, Inc. Methods and systems of extended reality environment interaction based on eye motions
US11320903B1 (en) 2020-10-20 2022-05-03 Rovi Guides, Inc. Methods and systems of extended reality environment interaction based on eye motions
US11392198B2 (en) 2020-10-20 2022-07-19 ROVl GUIDES, INC. Methods and systems of extended reality environment interaction based on eye motions
US11609629B2 (en) 2020-10-20 2023-03-21 Rovi Guides, Inc. Methods and systems of extended reality environment interaction based on eye motions
US11747896B2 (en) * 2020-10-20 2023-09-05 Rovi Guides, Inc. Methods and systems of extended reality environment interaction based on eye motions
US11782506B2 (en) 2020-10-20 2023-10-10 Rovi Guides, Inc. Methods and systems of extended reality environment interaction based on eye motions

Similar Documents

Publication Publication Date Title
US20200285379A1 (en) System for gaze interaction
US10394320B2 (en) System for gaze interaction
US10488919B2 (en) System for gaze interaction
US11573631B2 (en) System for gaze interaction
US11048873B2 (en) Emoji and canned responses
US20230384875A1 (en) Gesture detection, list navigation, and item selection using a crown and sensors
US11941191B2 (en) Button functionality
US11354015B2 (en) Adaptive user interfaces
US20180364802A1 (en) System for gaze interaction
US10540008B2 (en) System for gaze interaction
US10156967B2 (en) Device, method, and graphical user interface for tabbed and private browsing
US20190339863A1 (en) Devices, Methods, and Graphical User Interfaces for Keyboard Interface Functionalities
WO2018156912A1 (fr) Système d'interaction de regard
US10691330B2 (en) Device, method, and graphical user interface for force-sensitive gestures on the back of a device
EP3187976A2 (fr) Système d'interaction de regard
US20200341610A1 (en) Presenting user interfaces that update in response to detection of a hovering object
US20220365632A1 (en) Interacting with notes user interfaces
US11416136B2 (en) User interfaces for assigning and responding to user inputs

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18709897

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 20.12.2019)

122 Ep: pct application non-entry in european phase

Ref document number: 18709897

Country of ref document: EP

Kind code of ref document: A1