WO2011030212A1 - Method and apparatus for using generic software applications by means of ocular control and suitable methods of interaction - Google Patents

Method and apparatus for using generic software applications by means of ocular control and suitable methods of interaction Download PDF

Info

Publication number
WO2011030212A1
WO2011030212A1 PCT/IB2010/002271 IB2010002271W WO2011030212A1 WO 2011030212 A1 WO2011030212 A1 WO 2011030212A1 IB 2010002271 W IB2010002271 W IB 2010002271W WO 2011030212 A1 WO2011030212 A1 WO 2011030212A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
interactors
module
coordinates
data
Prior art date
Application number
PCT/IB2010/002271
Other languages
French (fr)
Inventor
Paolo Invernizzi
Original Assignee
Sr Labs S.R.L.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sr Labs S.R.L. filed Critical Sr Labs S.R.L.
Priority to ES10787877.9T priority Critical patent/ES2661554T3/en
Priority to NO10787877A priority patent/NO2476043T3/no
Priority to DK10787877.9T priority patent/DK2476043T3/en
Priority to EP10787877.9A priority patent/EP2476043B1/en
Priority to CN2010800394075A priority patent/CN102483650A/en
Priority to BR112012005342-6A priority patent/BR112012005342B1/en
Priority to US13/395,014 priority patent/US9372605B2/en
Priority to CA2773636A priority patent/CA2773636C/en
Publication of WO2011030212A1 publication Critical patent/WO2011030212A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G06F3/04842Selection of displayed objects or displayed text elements
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013Eye tracking input arrangements

Definitions

  • the present invention refers to the field of systems and methods for the control of generic software applications by means of eye-tracking devices, i.e. by using ocular movements.
  • the tracking of the ocular movements potentially offers the user the possibility of controlling the software of a personal computer by simply looking at the display thereof.
  • eye-tracking systems are generally limited to software made specifically for disabled users in which a low pointing precision is required. Indeed, the user should make his/her selection from a list of relatively large cells in which the effect of the error of accuracy is small.
  • eye-tracking systems for controlling the application, e.g. by means of the movement of the cursor which is updated many times per second by following the direction of the gaze, the error of accuracy makes the user's task very difficult, to whom a significant effort of concentration is required, which very soon generates fatigue.
  • Some solutions in use in the state of the art provide resolving the aforesaid problem by means of enlarging a part of the display of the personal computer, so as to improve the accuracy in the positioning of the cursor.
  • the functions wanted may be activated by pointing the gaze on icons arranged on strips of thin plastic-coated cardboard which are positioned on the vertical perimeter of the monitor: thereby the eye should select the functions external thereto before activating them on the icon or on the function wanted and then move the cursor which will be steered by the gaze towards the icons or applications wanted.
  • the directions of the mouse pointer are activated with the gaze on the screen view on the monitor of the eye-tracking device and once the direction is selected the effect is observed on the other PC; as soon as the gaze is removed from the PC wanted to be controlled, the pointer stops and the commands wanted may be selected. Thereby, the second PC connected may be steered with the eye-tracking system, by means of the emulation of the mouse.
  • One of the main requirements remains that of minimizing the cognitive effort required to manipulate the software by ensuring that the interface accepts "natural” inputs and responds "naturally” and is easily comprehensible.
  • the object of the present invention provides a new mapping of the "original" native interactors of the operating system of the personal computer (icons, etc.) in new interactors modified and made suitable to the selection mode by means of ocular control. Therefore the present invention establishes a sort of "direct channel", as the emulation of the cursor is overcome by the fact that the native interactors are replaced by those modified and adapted to the need to use the gaze as input system.
  • a Virtual Machine which permits to manage and create a virtual environment to carry out a further operating system with related software applications and which the user may use simultaneously to the one started, provides further advantages.
  • One of them is security: a virtual machine is completely isolated and independent and a sudden crash of the virtual machine does not involve the hosting operating system to crash; therefore restarting the computer is not required but only terminating the virtual machine and starting it again while avoiding damaging e.g. the file system.
  • the method according to the present invention is integrated in a communication suite, by using the virtual machine it is possible to simply, quickly and safely move (by means of suspension of the virtual machine) from using the classic PC applications, by means of the control techniques described below, to the communication suite designed specifically for being used via ocular control, thus overcoming the problems of the systems in the state of the art which provide rather complicated procedures for moving from one mode to the other.
  • the user may directly choose, by means of suitable shortcuts, to carry out some predefined applications inside the communication suite thus overcoming the problem of the possible direct execution from the operating system.
  • Fig. 1 shows the block diagram of the architecture of the method in accordance with the present invention.
  • Fig. 2 shows the flow diagram of the method according to the present invention.
  • Fig. 3 shows the flow diagram of the module relating to filtering the rough data coming from the eye-tracking device.
  • Fig. 4 shows the flow diagram relating to the Application Control Module.
  • Fig. 5 shows the flow diagram relating to the Coordinate Mapping Module.
  • Fig. 6 shows the flow diagram relating to the data recovery strategies relating to the interactors in the screen views.
  • Fig. 7 - 10 show, by way of non-limiting example, a possible implementation of the graphic interface of the application object of the present invention.
  • the object of the present invention consists of a method and an apparatus for using a generic operating system and generic software applications connected thereto, by means of ocular control.
  • a further object of the present invention consists of suitable methods of interaction developed by means of interaction techniques and an intuitive and easy-to-use user interface as described in the claims which form an integral part of the present description.
  • the method object of the present invention therefore depicts a possible implementation of an assistive technology, extremely innovative in terms of control of a generic operating system and of the applications connected thereto, based on the use of alternative and natural inputs, such as the gaze.
  • the apparatus object of the present invention comprises electronic means of data and of information processing, means for memorizing said data and information and user interfacing means.
  • Said electronic data and information processing means comprise a suitable control section, preferably based on at least a micro-processor, and may, e.g., be provided by a personal computer.
  • Said memorizing means preferably comprise hard disks and storage devices of flash type.
  • Said user interfacing means preferably comprise data visualising means, such as e.g. displays, monitors or analogous external output units and eye-tracking devices adapted to interpret the direction of the user's gaze.
  • Said micro-processor is preferably equipped with an operating system, with a suitable virtual environment by means of the use of a virtual machine and by a suitable software program which implements a method whose architecture, described in Figure 1 , comprises the following modules, in turn comprising a series of instructions adapted to performing a specific task: a filtering module 10 in which the coordinates of the user's gaze are processed so as to make the rough data coming from the used eye-tracking device more stable; a module, namely an Operating System/Applications Control 11 , responsible for controlling the operating system and the applications associated therewith and for running the developing application graphic interfaces, interface which contains the information about the interactors in the screen view and carries out the native action associated with the interactor fixed by the user at that moment, said interactors being the activatable elements in the interface: the icons, the pop-down menus, the check boxes etc., adapted to make the user carry out actions on the applications program to be controlled.
  • a filtering module 10 in which the coordinates of the user's
  • Said Operating System/Applications Control Module 11 is formed by two component sub-modules: a Controller Module 12 and a Client Module 13.
  • Said Controller Module 12 is in charge of managing the presentation of the interactors and of defining the native action associated with each of them and in turn comprises three further modules which interact with each other:
  • a Coordinate Mapping Module 12A which is in charge of carrying out new mapping of the coordinates relating to the screen views and to the interactors therein (different between Client and Controller);
  • an Interactor Managing Module 12B which is in charge of carrying out the comparison with the incoming gaze to define which interactors are fixed by the user and presents them, suitably and possibly modified, on the interface, e.g. on a side panel;
  • a Native Action Definition Module 12C which is in charge of defining the native action associated with each interactor of the Operating System and of sending it to said Client Module 13, thus making it available for successive processing.
  • Said Coordinate Mapping Module 12A in turn consists of two sub-modules which interact with each other: a Coordinates Translation Sub-Module 14 which carries out a translation of the coordinates relating to screen views and interactors and an Adaptive Calibration Sub-Module 15 which carries out a further re-adjustment of the coordinates by means of geometrical deformation of the plane obtained by comparing the information on the interactors which the user may select and the coordinates of the gaze coming from the eye-tracker, the results of the combined actions of these 2 modules is the one described above concerning the Coordinates Mapping Module 12A.
  • Said Client Module 13 is adapted to defining the position and the function of the interactors in the screen view and in turn comprises two further modules which interact with each other: the Interface Managing Module 13A which is in charge of analysing the screen views and sending the information relating thereto and to the present interactors, detected by means of different search methods, to the Controller Module 12; the Native Action Managing Module 13B which is in charge of receiving the information relating to the associated native action and of carrying it out as action on said operating system.
  • the Interface Managing Module 13A which is in charge of analysing the screen views and sending the information relating thereto and to the present interactors, detected by means of different search methods, to the Controller Module 12
  • the Native Action Managing Module 13B which is in charge of receiving the information relating to the associated native action and of carrying it out as action on said operating system.
  • a control user interface adapted to permit the user to control the operating system and the application programs associated therewith of said electronic processing means are visualised 20.
  • step e) The action to be carried out determined by the previous step is performed 23 and, possibly, said control user interface is suitably modified following the action itself. f) The sequence is repeated starting from step b) until the user decides to leave the application program which he/she is using.
  • a pre-filtering 30 of the rough data is performed and the non-valid samples are detected and eliminated using, e.g., statistical criteria.
  • the Interface Managing Module 13A analyses the screen views and sends 40 the information relating to the screen views and to the interactors in the current user interface to the Coordinate Mapping Module 12A.
  • the Coordinate Mapping Module 12A carries out new mapping of the coordinates relating to the screen views and to the interactors and carries out a comparison with the data relating to the gaze coming from the eye- tracker. Said new mapping allows the coordinates coming from the Client Module 13 relating to the position of the interactors in the screen views to be defined with respect to another system of axes having different origin than the one based on which the coordinates of the gazes coming from the eye-tracking device are defined. Once this operation has been carried out the comparison permits to understand which interactor is fixed by the user.
  • the Interactor Managing Module 12B draws again 42 the interactors present in the surroundings of the gazes detected by showing them suitably (e.g.
  • the Interactor Managing Module 12B then waits for the data relating to the gazes coming from the eye-tracker and for the selection of one of the interactors in the side panel.
  • the Native Action Definition Module 12C defines 43 the native action of the Operating System associated with each interactor (i.e. events which are simulated by the Client Module 13 and correspond to insertions of controls from the keyboard, simple click, drag & drop, etc.) and sends it to the Native Action Managing Module 13B.
  • the Native Action Managing Module 13B receives the information relating to the native action associated with the interactor selected and carries it out 44 as action on the Operating System (e.g. send character, mouse movement in specific position, etc.).
  • the Coordinate Translation Module 14 carries out 50 a translation of the coordinates relating to screen views and incoming interactors and sends these data to the Adaptive Calibration Module 15.
  • the Adaptive Calibration Module 15 carries out 51 a further re-adjustment of the coordinates, by means of geometrical deformation of the plane obtained by comparing the information on the interactors which the user may select and the coordinates of the gaze coming from the eye-tracker, and sends the information for updating the mapping to the Coordinate
  • the Interface Managing Module carries out the search of the interactors in the view screens continuously during the entire process described above, by means of the use of the steps described below and illustrated in Figure 6:
  • the Interface Managing Module 13A queries 60 the API (Application Programming Interfaces) of the accessibility functions to track the position and the functions of the different interactors in the page.
  • the Interface Managing Module 13A implements 61 an algorithm of recursive crossing of the diagram of windows in the page while trying to extract further information (in order to overcome the obstacle represented by insufficient information obtained by means of the accessibility API).
  • the Interface Managing Module 13A consults 62 a pre-formed database of supported applications from which it obtains the strategies for using various applications.
  • the database contains general information relating to how the various applications and the various application programs are made and heuristically obtains indications therefrom on how others of which nothing is known may be made.
  • the method described allows, e.g., a disabled user to use a personal computer equipped, e.g., with an operating system and with application programs such as Windows® and the Microsoft® Office® package.
  • the user is arranged in front of the eye-tracker connected to a monitor in which is visualised the screen view of the operating system/application wanted to be controlled by means of ocular movements.
  • the user fixes e.g., the Start icon on the Windows® application bar close to which is the icon for the Word® application and the Windows® bar: the gaze moves in specific surroundings for the intrinsic features of the eye-tracker. Due to this and to the possible and intrinsic error of accuracy of the eye-tracking device, what the user is fixing on and what his/her intention is may not be said with certainty.
  • all the interactors in the surrounding of the gaze e.g. Start button, Word® program icon, Windows®) bar
  • are shown in a side panel suitably for selection by means of ocular control (well spaced and of suitable sizes).
  • Such interactors are detected by means of suitable data recovery strategies (type, position, etc.) relating to the interactors in the screen view (accessibility API query, recursive crossing of windows diagram, database of predefined applications) and are shown in a "weighted” manner according to the order obtained by means of heuristic techniques (from the most probable to the least probable).
  • the user fixes the interactor of interest on the side panel and as the buttons are well spaced and of suitable sizes, there is no ambiguity with respect to the user's choice.
  • the button is therefore selected and the consequent action is carried out.
  • the user firstly, arranges him/herself in front of the eye-tracker connected to a monitor in which is visualised the screen view 70 of the operation system/application wanted to be controlled by means of ocular movements ( Figure 7).
  • the user fixes e.g., the Start icon on the Windows® application bar close to which is the icon for the Word® application and the Windows® bar: the gaze 71 moves in specific surroundings for the intrinsic features of the eye-tracker. Due to this and to the possible and intrinsic error of accuracy of the eye-tracking device, what the user is fixing on and what his/her intention is may not be said with certainty (Figure 7).
  • All the interactors in the surrounding of the gaze 71 are shown in a side panel 72, suitably for selection by means of ocular control (well spaced and of suitable sizes).
  • Such interactors are detected by means of suitable data recovery strategies (type, position, etc.) relating to the interactors in the screen view (accessibility API query, recursive crossing of windows diagram, database of predefined applications) and are shown in a "weighted" manner according to the order obtained by means of heuristic techniques (from the most probable to the least probable) ( Figure 8).
  • the user fixes the interactor of interest 73 on the side panel 72: as the buttons are well spaced and of suitable sizes, there is no ambiguity with respect to the choice of the user ( Figure 9).
  • the button is selected and the consequent action is carried out.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)
  • Position Input By Displaying (AREA)
  • Prostheses (AREA)
  • Stored Programmes (AREA)

Abstract

The method and apparatus object of this invention refer to a system for using generic software applications by means of ocular control characterized by simple interaction techniques which minimize the cognitive effort of the user required to manipulate the software and which permit to overcome the problems resulting from the intrinsic problem of accuracy of eye-tracking systems. Indeed, such technique does not provide using the pointer of the mouse moved by means of the gaze to control the various software applications but to use a separate application which through the use of suitable interaction techniques is comfortable and does not involve an increased effort of concentration by the user. An attempt has indeed been made to simplify the process of interaction between the user and machine also by means of the use of visual feedback which allows the same operations of the mouse to be performed by the user without the typical user frustration due to the problems of accuracy of the eye-tracking device.

Description

METHOD AND APPARATUS FOR USING GENERIC SOFTWARE APPLICATIONS BY MEANS OF OCULAR CONTROL AND SUITABLE METHODS OF INTERACTION
Field of the invention
The present invention refers to the field of systems and methods for the control of generic software applications by means of eye-tracking devices, i.e. by using ocular movements.
Not having the opportunity to use the mouse for access to the software applications, the user should take advantage of the techniques which permit him/her to carry out, by means of ocular movements, the same operations which are possible to carry out with the mouse. To this end, techniques and contrivances have been developed which permit the user to significantly decrease the cognitive effort to be performed.
State of the art
The tracking of the ocular movements potentially offers the user the possibility of controlling the software of a personal computer by simply looking at the display thereof.
However, there are problems which frustrate the user during the use of the tracking systems of ocular movements - or eye-tracking systems - for controlling the software of a personal computer. In particular, the eyes should simultaneously behave as input and output devices, i.e. they should explore and carry out true actions. Again, the eye-tracking devices suffer intrinsic problems of accuracy in measuring the position of the gaze on the screen. Therefore, the accuracy of the sensor is always affected by an error (the difference between the current position of the cursor on the display compared with the position of the cursor wanted by the user) and this does not make controlling most of the software applications easy, given that many software programs require an accuracy of some millimeters, and others also require an accuracy assessable in terms of pixels.
Thus the use of eye-tracking systems is generally limited to software made specifically for disabled users in which a low pointing precision is required. Indeed, the user should make his/her selection from a list of relatively large cells in which the effect of the error of accuracy is small. Instead, in the field of software applications of common use, the employment of eye-tracking systems for controlling the application, e.g. by means of the movement of the cursor which is updated many times per second by following the direction of the gaze, the error of accuracy makes the user's task very difficult, to whom a significant effort of concentration is required, which very soon generates fatigue.
Any attempt by the user to correct the error by re-directing his/her gaze towards the position where the cursor should be positioned does nothing more than making the use of the cursor increasingly problematic and tiring.
Some solutions in use in the state of the art provide resolving the aforesaid problem by means of enlarging a part of the display of the personal computer, so as to improve the accuracy in the positioning of the cursor.
However, this approach is insufficient because the error of compensation also remains in these cases, thus continuing to frustrate the user, and moreover implies an increased complexity of the interface which may create confusion and distraction.
Other solutions currently available imply the appearance of "off-screen" menus which permit the user to select which action to carry out by emulating the different operations which may be carried out with the mouse (single click, double click, right click, drag and drop).
Again, in some cases, the functions wanted may be activated by pointing the gaze on icons arranged on strips of thin plastic-coated cardboard which are positioned on the vertical perimeter of the monitor: thereby the eye should select the functions external thereto before activating them on the icon or on the function wanted and then move the cursor which will be steered by the gaze towards the icons or applications wanted.
In further other cases, it is also possible to steer a second PC, connected with the eye-tracking system, whose monitor should be installed beside the monitor of the eye-tracker to give the user the opportunity to easily observe the control of the mouse over the applications.
The directions of the mouse pointer are activated with the gaze on the screen view on the monitor of the eye-tracking device and once the direction is selected the effect is observed on the other PC; as soon as the gaze is removed from the PC wanted to be controlled, the pointer stops and the commands wanted may be selected. Thereby, the second PC connected may be steered with the eye-tracking system, by means of the emulation of the mouse.
In principle, we may assert that, contrarily to what has been implemented to date, it would be desirable to have a display on which the elements depicted do not interfere too much with the normal use of the software for the following reasons: the attention of the user declines exponentially with the increasing number of elements depicted on the screen, the user normally has little familiarity with the eye-tracking devices and moreover may be affected by cognitive disabilities such as to make the use of an eye-tracking device prohibitive.
For the aforesaid reasons, it is the object of the present invention to provide a method for controlling a personal computer by means of an eye-tracking system which overcomes the drawbacks listed above.
One of the main requirements remains that of minimizing the cognitive effort required to manipulate the software by ensuring that the interface accepts "natural" inputs and responds "naturally" and is easily comprehensible.
For this reason it will be necessary to develop an interface which uses interaction techniques which are not stressful for the user. Given that many of the inputs are unintentional, the system should interpret them correctly without producing unrequested responses caused by involuntary actions. Such system therefore should be capable of distinguishing between the true will of the user while letting him/her observe the interface peacefully if it is not his/her intention to give a command; on the other hand the user should be capable of assessing what the current status of the system is, so as to realize if his/her intentions were interpreted properly, to avoid the execution of involuntary commands.
It should also be underlined that while the systems of the state of the art control operating systems by means of the emulation of the mouse via ocular control (conversion of the movement of the gaze into movement of the cursor), the object of the present invention provides a new mapping of the "original" native interactors of the operating system of the personal computer (icons, etc.) in new interactors modified and made suitable to the selection mode by means of ocular control. Therefore the present invention establishes a sort of "direct channel", as the emulation of the cursor is overcome by the fact that the native interactors are replaced by those modified and adapted to the need to use the gaze as input system.
Also, the use of a Virtual Machine, which permits to manage and create a virtual environment to carry out a further operating system with related software applications and which the user may use simultaneously to the one started, provides further advantages. One of them is security: a virtual machine is completely isolated and independent and a sudden crash of the virtual machine does not involve the hosting operating system to crash; therefore restarting the computer is not required but only terminating the virtual machine and starting it again while avoiding damaging e.g. the file system.
As the method according to the present invention is integrated in a communication suite, by using the virtual machine it is possible to simply, quickly and safely move (by means of suspension of the virtual machine) from using the classic PC applications, by means of the control techniques described below, to the communication suite designed specifically for being used via ocular control, thus overcoming the problems of the systems in the state of the art which provide rather complicated procedures for moving from one mode to the other.
At the same time, the user may directly choose, by means of suitable shortcuts, to carry out some predefined applications inside the communication suite thus overcoming the problem of the possible direct execution from the operating system.
Brief description of the figures
Fig. 1 shows the block diagram of the architecture of the method in accordance with the present invention.
Fig. 2 shows the flow diagram of the method according to the present invention. Fig. 3 shows the flow diagram of the module relating to filtering the rough data coming from the eye-tracking device.
Fig. 4 shows the flow diagram relating to the Application Control Module.
Fig. 5 shows the flow diagram relating to the Coordinate Mapping Module.
Fig. 6 shows the flow diagram relating to the data recovery strategies relating to the interactors in the screen views.
Fig. 7 - 10 show, by way of non-limiting example, a possible implementation of the graphic interface of the application object of the present invention.
Summary of the invention
The object of the present invention consists of a method and an apparatus for using a generic operating system and generic software applications connected thereto, by means of ocular control. A further object of the present invention consists of suitable methods of interaction developed by means of interaction techniques and an intuitive and easy-to-use user interface as described in the claims which form an integral part of the present description.
The method object of the present invention therefore depicts a possible implementation of an assistive technology, extremely innovative in terms of control of a generic operating system and of the applications connected thereto, based on the use of alternative and natural inputs, such as the gaze.
Detailed description of the invention
In a preferred embodiment of the present invention, the apparatus object of the present invention comprises electronic means of data and of information processing, means for memorizing said data and information and user interfacing means.
Said electronic data and information processing means comprise a suitable control section, preferably based on at least a micro-processor, and may, e.g., be provided by a personal computer.
Said memorizing means preferably comprise hard disks and storage devices of flash type. Said user interfacing means preferably comprise data visualising means, such as e.g. displays, monitors or analogous external output units and eye-tracking devices adapted to interpret the direction of the user's gaze.
Said micro-processor is preferably equipped with an operating system, with a suitable virtual environment by means of the use of a virtual machine and by a suitable software program which implements a method whose architecture, described in Figure 1 , comprises the following modules, in turn comprising a series of instructions adapted to performing a specific task: a filtering module 10 in which the coordinates of the user's gaze are processed so as to make the rough data coming from the used eye-tracking device more stable; a module, namely an Operating System/Applications Control 11 , responsible for controlling the operating system and the applications associated therewith and for running the developing application graphic interfaces, interface which contains the information about the interactors in the screen view and carries out the native action associated with the interactor fixed by the user at that moment, said interactors being the activatable elements in the interface: the icons, the pop-down menus, the check boxes etc., adapted to make the user carry out actions on the applications program to be controlled.
Said Operating System/Applications Control Module 11 is formed by two component sub-modules: a Controller Module 12 and a Client Module 13.
Said Controller Module 12 is in charge of managing the presentation of the interactors and of defining the native action associated with each of them and in turn comprises three further modules which interact with each other:
a Coordinate Mapping Module 12A which is in charge of carrying out new mapping of the coordinates relating to the screen views and to the interactors therein (different between Client and Controller);
an Interactor Managing Module 12B which is in charge of carrying out the comparison with the incoming gaze to define which interactors are fixed by the user and presents them, suitably and possibly modified, on the interface, e.g. on a side panel;
a Native Action Definition Module 12C which is in charge of defining the native action associated with each interactor of the Operating System and of sending it to said Client Module 13, thus making it available for successive processing.
Said Coordinate Mapping Module 12A in turn consists of two sub-modules which interact with each other: a Coordinates Translation Sub-Module 14 which carries out a translation of the coordinates relating to screen views and interactors and an Adaptive Calibration Sub-Module 15 which carries out a further re-adjustment of the coordinates by means of geometrical deformation of the plane obtained by comparing the information on the interactors which the user may select and the coordinates of the gaze coming from the eye-tracker, the results of the combined actions of these 2 modules is the one described above concerning the Coordinates Mapping Module 12A.
Said Client Module 13 is adapted to defining the position and the function of the interactors in the screen view and in turn comprises two further modules which interact with each other: the Interface Managing Module 13A which is in charge of analysing the screen views and sending the information relating thereto and to the present interactors, detected by means of different search methods, to the Controller Module 12; the Native Action Managing Module 13B which is in charge of receiving the information relating to the associated native action and of carrying it out as action on said operating system.
In reference to Figure 2, there is shown a diagram which depicts the operation of the modules mentioned above and the interconnections with each other by illustrating the steps of the method according to the present invention:
a) On said data visualising means associated with said data processing means, a control user interface adapted to permit the user to control the operating system and the application programs associated therewith of said electronic processing means are visualised 20.
b) The coordinates of the user's gaze in the form of rough data, i.e. of samples relating to the coordinates of the gaze of the two separate eyes, which are subject to strong oscillations, are detected by the eye-tracking device 21 , comprised in said user interfacing means. These oscillations generally occur about a certain position, but there are also some gazes which are totally erroneous and should be eliminated by means of a filtering operation. c) Said rough data are filtered 22 so as to make them stable and suitable for providing indications on the fixations of the user, i.e. on the number of gazes of the user within certain surroundings.
d) The filtered data expressed in x, y coordinates of the fixed point are sent 23 to the Operating System/Applications Control Module 11 which processes them by defining the action to be carried out and the modifications to be made on said user interface.
e) The action to be carried out determined by the previous step is performed 23 and, possibly, said control user interface is suitably modified following the action itself. f) The sequence is repeated starting from step b) until the user decides to leave the application program which he/she is using.
The filtering procedure of rough data according to step c) is carried out according to the sequence indicated below and illustrated in Figure 3:
g) A pre-filtering 30 of the rough data is performed and the non-valid samples are detected and eliminated using, e.g., statistical criteria.
h) The coordinates of the gaze of the right eye and of the left eye are combined 31 in a point deemed probable of being the one fixed by the user. i) The average and the standard deviation are calculated 32 with respect to the number of samples wanted (defined based on the number of samples which were decided a priori to take into consideration).
j) The tolerances are defined 33 for the evaluations.
k) The acceptable data are discriminated 34 with respect to those to be rejected based on what was established in the previous steps i) and j).
The filtered data sent to the Operating System/Applications Control Module 11 are processed according to the sequence indicated below and illustrated in Figure 4:
I) The Interface Managing Module 13A analyses the screen views and sends 40 the information relating to the screen views and to the interactors in the current user interface to the Coordinate Mapping Module 12A.
m) The Coordinate Mapping Module 12A carries out new mapping of the coordinates relating to the screen views and to the interactors and carries out a comparison with the data relating to the gaze coming from the eye- tracker. Said new mapping allows the coordinates coming from the Client Module 13 relating to the position of the interactors in the screen views to be defined with respect to another system of axes having different origin than the one based on which the coordinates of the gazes coming from the eye-tracking device are defined. Once this operation has been carried out the comparison permits to understand which interactor is fixed by the user. n) The Interactor Managing Module 12B draws again 42 the interactors present in the surroundings of the gazes detected by showing them suitably (e.g. on a side panel of the specific user interface of the program) by using heuristic techniques to decide the order of appearance by determining which interactor will be most probably selected. This permits to show each interactor in "weighted" manner, from the most to the least probable. The Interactor Managing Module 12B then waits for the data relating to the gazes coming from the eye-tracker and for the selection of one of the interactors in the side panel.
o) The Native Action Definition Module 12C defines 43 the native action of the Operating System associated with each interactor (i.e. events which are simulated by the Client Module 13 and correspond to insertions of controls from the keyboard, simple click, drag & drop, etc.) and sends it to the Native Action Managing Module 13B.
p) The Native Action Managing Module 13B receives the information relating to the native action associated with the interactor selected and carries it out 44 as action on the Operating System (e.g. send character, mouse movement in specific position, etc.).
The process of mapping the coordinates again according to step m) of the sequence illustrated in Figure 4 occurs according to the sequence indicated below and illustrated in Figure 5:
q) The Coordinate Translation Module 14 carries out 50 a translation of the coordinates relating to screen views and incoming interactors and sends these data to the Adaptive Calibration Module 15.
r) The Adaptive Calibration Module 15 carries out 51 a further re-adjustment of the coordinates, by means of geometrical deformation of the plane obtained by comparing the information on the interactors which the user may select and the coordinates of the gaze coming from the eye-tracker, and sends the information for updating the mapping to the Coordinate
Translation Module 14.
The Interface Managing Module carries out the search of the interactors in the view screens continuously during the entire process described above, by means of the use of the steps described below and illustrated in Figure 6:
s) The Interface Managing Module 13A queries 60 the API (Application Programming Interfaces) of the accessibility functions to track the position and the functions of the different interactors in the page. t) The Interface Managing Module 13A implements 61 an algorithm of recursive crossing of the diagram of windows in the page while trying to extract further information (in order to overcome the obstacle represented by insufficient information obtained by means of the accessibility API).
u) The Interface Managing Module 13A consults 62 a pre-formed database of supported applications from which it obtains the strategies for using various applications. The database contains general information relating to how the various applications and the various application programs are made and heuristically obtains indications therefrom on how others of which nothing is known may be made.
The method described allows, e.g., a disabled user to use a personal computer equipped, e.g., with an operating system and with application programs such as Windows® and the Microsoft® Office® package.
The actions carried out by the user are described below in detail:
The user is arranged in front of the eye-tracker connected to a monitor in which is visualised the screen view of the operating system/application wanted to be controlled by means of ocular movements.
The user fixes, e.g., the Start icon on the Windows® application bar close to which is the icon for the Word® application and the Windows® bar: the gaze moves in specific surroundings for the intrinsic features of the eye-tracker. Due to this and to the possible and intrinsic error of accuracy of the eye-tracking device, what the user is fixing on and what his/her intention is may not be said with certainty. To obviate this problem, all the interactors in the surrounding of the gaze (e.g. Start button, Word® program icon, Windows®) bar) are shown in a side panel, suitably for selection by means of ocular control (well spaced and of suitable sizes). Such interactors are detected by means of suitable data recovery strategies (type, position, etc.) relating to the interactors in the screen view (accessibility API query, recursive crossing of windows diagram, database of predefined applications) and are shown in a "weighted" manner according to the order obtained by means of heuristic techniques (from the most probable to the least probable).
The user fixes the interactor of interest on the side panel and as the buttons are well spaced and of suitable sizes, there is no ambiguity with respect to the user's choice. The button is therefore selected and the consequent action is carried out. In greater detail and in reference to accompanying figures 7-10, the user, firstly, arranges him/herself in front of the eye-tracker connected to a monitor in which is visualised the screen view 70 of the operation system/application wanted to be controlled by means of ocular movements (Figure 7).
The user fixes, e.g., the Start icon on the Windows® application bar close to which is the icon for the Word® application and the Windows® bar: the gaze 71 moves in specific surroundings for the intrinsic features of the eye-tracker. Due to this and to the possible and intrinsic error of accuracy of the eye-tracking device, what the user is fixing on and what his/her intention is may not be said with certainty (Figure 7).
All the interactors in the surrounding of the gaze 71 (e.g. Start button, Word® icon, Windows®) bar) are shown in a side panel 72, suitably for selection by means of ocular control (well spaced and of suitable sizes). Such interactors are detected by means of suitable data recovery strategies (type, position, etc.) relating to the interactors in the screen view (accessibility API query, recursive crossing of windows diagram, database of predefined applications) and are shown in a "weighted" manner according to the order obtained by means of heuristic techniques (from the most probable to the least probable) (Figure 8).
The user fixes the interactor of interest 73 on the side panel 72: as the buttons are well spaced and of suitable sizes, there is no ambiguity with respect to the choice of the user (Figure 9). The button is selected and the consequent action is carried out.
After the selection of the Start button from the interactors panel, the action is consequently carried out (Figure 10), or the window is opened 74 relating to the request. The interactors panel 72 is emptied while waiting to be filled again with new interactors following the successive gazes of the user.

Claims

1. Method for using generic software applications by means of ocular control comprising the following steps:
a) visualising (20) on suitable given visualising means associated to suitable data processing means a control user interface for allowing a user to control the operating system, and the associated application programs of said data processing means;
b) detecting (21 ) by means of an eye-tracking device 21 (eye-tracker), comprised in suitable user interfacing means associated to said data processing means, the coordinates of the user's gaze in the form of rough data, namely of samples concerning the coordinates of the gaze of each user's eye;
c) filtering (22) said rough data in order to make them stable and suitable to provide indications on the fixations of the user, namely on the number of gazes of the user within certain surroundings;
d) sending (23) the filtered data, expressed in x, y coordinates of the fixed point, to the Operating System Control Module/Applications (11 ) processing them by defining the action to be carried out and the changes to be introduced on said control user interface; said Operating System/Applications Control Module (11 ) comprising a set of instructions to control the operating system and the related applications and to run the developing application graphic interface containing the information about the interactors present in the screen view and carrying out the native action associated to the interactor fixed by the user at that moment;
e) carrying out the action determined by the previous step (23) and, in case it is necessary, said control user interface is suitably modified according to the occurred action;
f) repeating the sequence starting from step b) until the user decides to leave the application program he is using.
2. Method according to Claim 1 , wherein the filtering procedure of rough data according to the previous step c) is carried out by means of the following steps:
g) pre-filtering (30) of rough data by detecting and eliminating the non valid samples using statistical criteria; h) combining (31) the coordinates of the gaze of the right eye and of the left eye in a point which is probably the one fixed by the user;
i) calculating (32) the average and the standard deviation if compared to the desired number of samples (defined according to the number of samples which were a priori taken into consideration);
j) defining (33) the tolerances for the evaluations;
k) selecting (34) the acceptable data which are discriminated if compared to those to be rejected according to what stated in the previous i) and j) steps.
3. Method according to Claims 1-2 wherein the processing of data received from said Operating System/Applications Control Module (11 ) according to step d) is carried out according to the following sequence of steps:
I) analysing, by means of an Interface Managing Module (13A) the screen views of the application program or of the operating system that the user wants to control by ocular control and sending (40) a Coordinate Mapping Module (12A) of the information concerning the screen views and the interactors present in the actual user interface; said Interface Managing Module (13A) comprising a set of instructions able to analyse the screen views and to detect the information concerning them and the relative interactors; said Coordinate Mapping Module (12A) comprising a set of instructions able to carry out a new mapping of the coordinates relating to the screen views and the relative interactors;
m) new mapping (41 ) of the coordinates concerning the screen views and the relative interactors by means of said Coordinate Mapping Module (12A) by means of a comparison with the data relating to the gaze of the user coming from the eye-tracker, said new mapping allowing to define the coordinates coming from the Client Module (13) regarding the position of the interactors in the screen views, if compared to another system of axes having a different origin from the one defining the coordinates of the gazes coming from the eye-tracker; said Client Module (13) comprising a set of instructions able to define the position and the function of the present interactors in the actual screen view;
n) the Interactor Managing Module (12B) comprising a set of instructions able to carry out the control of the coming "gazes" in order to define which interactors are defined by the user and to suitably show said interactors on the interface, draws again (42) the interactors present in the surroundings of the detected gazes suitably showing them, possibly modified on the interface, by using heuristic techniques to decide the order of appearance by determining which interactor will be most probably selected. Said Interactor Managing Module (12B) waits then for the data relative to the gazes coming from the eye-tracker and for the selection of one of the interactors present in the side panel;
o) the Native Action Definition Module (12C), comprising a set of instructions able to define the native action associated to each interactor of the Operating System and to make it available for successive processing, defines (43) the native action of the Operating System associated to each interactor (namely events corresponding to the insertion of controls from the keyboard, simple click, drag & drop, etc.) and sends it to the Native Action Managing Module (13B) comprising a set of instructions able to receive the information concerning to the native action associated to the selected interactor and to carry out it as action on said operating system;
p) said Native Action Managing Module (13B) receives the information relative to the native action associated to the selected interactor (44) as action on the Operating System.
4. Method according to Claim 3, wherein the process of coordinates re- mapping according to step m) is carried out according to the following sequence of steps:
q) the Coordinate Translation Module (14), comprising a set of instructions able to carry out a translation of the coordinates relative to screen views and interactors, carries out (50) a translation of the coordinates relative to the information concerning the screen views and the coming interactors and sends these data to the Adaptive Calibration Module (15);
r) said Adaptive Calibration Module (15), comprising a set of instruction able to carry out a further re-adjustment of the coordinates by means of geometrical deformation of the plane obtained by comparing the information on the interactors that the user can select and the coordinates of the gaze coming from the eye- tracker, re-adjusts (51 ) the coordinates and sends the information for updating the mapping to said Coordinates Translation Module (14).
5. Method according to Claims 1-4, wherein said Interface Managing Module ( 3A) carries out the search of the interactors present in the screen views, in a continuous way, by means of the following sequence of steps:
s) said Interface managing Module (13A) questions (60) the API (Application Programming Interfaces) of the accessibility functions to track the position and the functions of the different interactors present in the page;
t) said Interface Managing Module (13A) implements (61 ) an algorithm of recursive crossing of the diagram of windows present in the page trying to extract further information in order to overcome the obstacle represented by possibly insufficient information obtained by means of the API of accessibility;
u) the Interface Managing Module (13A) consults (62) a pre-formed database of supported applications from which it obtains the strategies for using various applications, said database containing general information concerning how said applications and said application programs are made.
6. Apparatus for using generic software applications by means of ocular control comprising electronic processing means of data and information, means for memorizing said data and information and user interface means wherein said electronic processing means of data and information comprise a suitable control selection based on at least a micro-processor, and said user interfacing means comprise display, monitor or analogous output units and eye-tracking devices able to interpret the direction of the gaze of the user.
7. Apparatus according to Claim 1 , wherein said electronic processing means of data and information consist of a personal computer.
8. Apparatus according to Claims 1-2, wherein said micro-processor comprises an operating system, a suitable virtual environment by using a virtual machine and a suitable software program.
9. Apparatus according to Claim 3, wherein said suitable software program implements the method according to Claims 1-5.
10. Computer program comprising means for programming a code able to carry out the steps of the method according to Claims 1-5 wherein said program is carried out by means of a computer.
11. Computer-readable recording support having a program recorded on it, said" readable recording support comprising coding means for a computer program adapted to implement all the steps of claims 1-5, when said computer program is run on the computer.
PCT/IB2010/002271 2009-09-11 2010-09-13 Method and apparatus for using generic software applications by means of ocular control and suitable methods of interaction WO2011030212A1 (en)

Priority Applications (8)

Application Number Priority Date Filing Date Title
ES10787877.9T ES2661554T3 (en) 2009-09-11 2010-09-13 Method and apparatus for using generic software applications by means of an eye control and appropriate interaction methods
NO10787877A NO2476043T3 (en) 2009-09-11 2010-09-13
DK10787877.9T DK2476043T3 (en) 2009-09-11 2010-09-13 Method and Device for Using Generic Software Applications by Eye Management and Appropriate Methods of Interaction
EP10787877.9A EP2476043B1 (en) 2009-09-11 2010-09-13 Method and apparatus for using generic software applications by means of ocular control and suitable methods of interaction
CN2010800394075A CN102483650A (en) 2009-09-11 2010-09-13 Method and apparatus for using generic software applications by means of ocular control and suitable methods of interaction
BR112012005342-6A BR112012005342B1 (en) 2009-09-11 2010-09-13 METHOD FOR CONTROLLING THE OPERATION OF AN OPERATING SYSTEM AND ASSOCIATED APPLICATIONS PROGRAM
US13/395,014 US9372605B2 (en) 2009-09-11 2010-09-13 Method and apparatus for controlling the operation of an operating system and application programs by ocular control
CA2773636A CA2773636C (en) 2009-09-11 2010-09-13 Method and apparatus for using generic software applications by ocular control and suitable methods of interaction

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
ITFI2009A000198 2009-09-11
ITFI2009A000198A IT1399456B1 (en) 2009-09-11 2009-09-11 METHOD AND APPARATUS FOR THE USE OF GENERIC SOFTWARE APPLICATIONS THROUGH EYE CONTROL AND INTERACTION METHODS IS APPROPRIATE.

Publications (1)

Publication Number Publication Date
WO2011030212A1 true WO2011030212A1 (en) 2011-03-17

Family

ID=42102373

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2010/002271 WO2011030212A1 (en) 2009-09-11 2010-09-13 Method and apparatus for using generic software applications by means of ocular control and suitable methods of interaction

Country Status (10)

Country Link
US (1) US9372605B2 (en)
EP (1) EP2476043B1 (en)
CN (1) CN102483650A (en)
CA (1) CA2773636C (en)
DK (1) DK2476043T3 (en)
ES (1) ES2661554T3 (en)
IT (1) IT1399456B1 (en)
NO (1) NO2476043T3 (en)
PT (1) PT2476043T (en)
WO (1) WO2011030212A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ITFI20120165A1 (en) * 2012-08-08 2014-02-09 Sr Labs S R L INTERACTIVE EYE CONTROL MULTIMEDIA SYSTEM FOR ACTIVE AND PASSIVE TRACKING
WO2013173148A3 (en) * 2012-05-18 2016-06-09 Microsoft Corporation Interaction and management of devices using gaze detection

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014015521A1 (en) * 2012-07-27 2014-01-30 Nokia Corporation Multimodal interaction with near-to-eye display
US20140049462A1 (en) * 2012-08-20 2014-02-20 Google Inc. User interface element focus based on user's gaze
CN103870097A (en) * 2012-12-12 2014-06-18 联想(北京)有限公司 Information processing method and electronic equipment
US9342145B2 (en) * 2013-01-22 2016-05-17 Kabushiki Kaisha Toshiba Cursor control
CN104007909B (en) * 2013-02-25 2019-03-01 腾讯科技(深圳)有限公司 Page automatic adjusting method and device
US9829975B2 (en) 2013-08-27 2017-11-28 Auckland Uniservices Limited Gaze-controlled interface method and system
JP6115418B2 (en) 2013-09-11 2017-04-19 富士通株式会社 Information processing apparatus, method, and program
US9310886B2 (en) * 2014-02-13 2016-04-12 Lenovo (Singapore) Pte. Ltd. Transient message display control
US9804753B2 (en) * 2014-03-20 2017-10-31 Microsoft Technology Licensing, Llc Selection using eye gaze evaluation over time
WO2015164467A1 (en) * 2014-04-23 2015-10-29 Abbott Medical Optics Inc. Medical device data filtering for real time display
US11430414B2 (en) * 2019-10-17 2022-08-30 Microsoft Technology Licensing, Llc Eye gaze control of magnification user interface
DE102022110291B3 (en) 2022-04-27 2023-11-02 Universität Stuttgart, Körperschaft Des Öffentlichen Rechts Computer-implemented method and system for hands-free selection of a control on a screen
CN115122342B (en) * 2022-09-02 2022-12-09 北京壹点灵动科技有限公司 Software system for controlling robot and control method of robot
CN117369649B (en) * 2023-12-05 2024-03-26 山东大学 Virtual reality interaction system and method based on proprioception

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6323884B1 (en) * 1999-03-31 2001-11-27 International Business Machines Corporation Assisting user selection of graphical user interface elements
US20050047629A1 (en) * 2003-08-25 2005-03-03 International Business Machines Corporation System and method for selectively expanding or contracting a portion of a display using eye-gaze tracking
WO2006045843A1 (en) * 2004-10-29 2006-05-04 Sr Labs S.R.L. Method and system of visualisation, processing and integrated analysis of medical images
WO2007017500A1 (en) * 2005-08-10 2007-02-15 Sr Labs S.R.L. Method and apparatus for secure insertion of an access code using an eye-tracking device

Family Cites Families (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4950069A (en) * 1988-11-04 1990-08-21 University Of Virginia Eye movement detector with improved calibration and speed
US5471542A (en) * 1993-09-27 1995-11-28 Ragland; Richard R. Point-of-gaze tracker
GB9518477D0 (en) * 1995-09-09 1995-11-08 Strachan John S Point of observation tracking system
JPH09114543A (en) * 1995-10-02 1997-05-02 Xybernaut Corp Handfree computer system
US5912721A (en) * 1996-03-13 1999-06-15 Kabushiki Kaisha Toshiba Gaze detection apparatus and its method as well as information display apparatus
US6373961B1 (en) * 1996-03-26 2002-04-16 Eye Control Technologies, Inc. Eye controllable screen pointer
US5731805A (en) * 1996-06-25 1998-03-24 Sun Microsystems, Inc. Method and apparatus for eyetrack-driven text enlargement
US5898423A (en) * 1996-06-25 1999-04-27 Sun Microsystems, Inc. Method and apparatus for eyetrack-driven captioning
US5886683A (en) * 1996-06-25 1999-03-23 Sun Microsystems, Inc. Method and apparatus for eyetrack-driven information retrieval
US5638176A (en) * 1996-06-25 1997-06-10 International Business Machines Corporation Inexpensive interferometric eye tracking system
US5850211A (en) * 1996-06-26 1998-12-15 Sun Microsystems, Inc. Eyetrack-driven scrolling
US6152563A (en) * 1998-02-20 2000-11-28 Hutchinson; Thomas E. Eye gaze direction tracker
US6204828B1 (en) * 1998-03-31 2001-03-20 International Business Machines Corporation Integrated gaze/manual cursor positioning system
GB9823977D0 (en) * 1998-11-02 1998-12-30 Scient Generics Ltd Eye tracking method and apparatus
US6577329B1 (en) * 1999-02-25 2003-06-10 International Business Machines Corporation Method and system for relevance feedback through gaze tracking and ticker interfaces
US6724402B1 (en) * 2000-01-27 2004-04-20 David R. Baquero Method of launching computer programs within a graphical user interface
US6456262B1 (en) * 2000-05-09 2002-09-24 Intel Corporation Microdisplay with eye gaze detection
US6603491B2 (en) * 2000-05-26 2003-08-05 Jerome H. Lemelson System and methods for controlling automatic scrolling of information on a display or screen
US6601021B2 (en) * 2000-12-08 2003-07-29 Xerox Corporation System and method for analyzing eyetracker data
US6578962B1 (en) * 2001-04-27 2003-06-17 International Business Machines Corporation Calibration-free eye gaze tracking
US6886137B2 (en) * 2001-05-29 2005-04-26 International Business Machines Corporation Eye gaze control of dynamic information presentation
US7913183B2 (en) * 2002-10-08 2011-03-22 Microsoft Corporation System and method for managing software applications in a graphical user interface
US7281248B2 (en) * 2002-11-19 2007-10-09 Microsoft Corporation Virtualized and realized user interface controls
US7206022B2 (en) * 2002-11-25 2007-04-17 Eastman Kodak Company Camera system with eye monitoring
US6637883B1 (en) * 2003-01-23 2003-10-28 Vishwas V. Tengshe Gaze tracking system and method
US7401920B1 (en) * 2003-05-20 2008-07-22 Elbit Systems Ltd. Head mounted eye tracking and display system
US7365738B2 (en) * 2003-12-02 2008-04-29 International Business Machines Corporation Guides and indicators for eye movement monitoring systems
ES2535364T3 (en) * 2004-06-18 2015-05-08 Tobii Ab Eye control of computer equipment
EP1679577A1 (en) * 2005-01-10 2006-07-12 Tobii Technology AB Adaptive display of eye controllable objects
WO2007107949A1 (en) * 2006-03-23 2007-09-27 Koninklijke Philips Electronics N.V. Hotspots for eye track control of image manipulation
US8225229B2 (en) * 2006-11-09 2012-07-17 Sony Mobile Communications Ab Adjusting display brightness and/or refresh rates based on eye tracking
US20080270910A1 (en) * 2007-04-30 2008-10-30 Lukasik Derek J User selection of a remote session
US7556377B2 (en) * 2007-09-28 2009-07-07 International Business Machines Corporation System and method of detecting eye fixations using adaptive thresholds
ITFI20080049A1 (en) * 2008-03-12 2009-09-13 Sr Labs Srl APPARATUS FOR THE CREATION, RESCUE AND FORMAT OF TEXTUAL DOCUMENTS THROUGH EYE CONTROL AND ASSOCIATED METHOD BASED ON THE OPTIMIZED POSITIONING OF THE CURSOR.
US20100182232A1 (en) * 2009-01-22 2010-07-22 Alcatel-Lucent Usa Inc. Electronic Data Input System

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6323884B1 (en) * 1999-03-31 2001-11-27 International Business Machines Corporation Assisting user selection of graphical user interface elements
US20050047629A1 (en) * 2003-08-25 2005-03-03 International Business Machines Corporation System and method for selectively expanding or contracting a portion of a display using eye-gaze tracking
WO2006045843A1 (en) * 2004-10-29 2006-05-04 Sr Labs S.R.L. Method and system of visualisation, processing and integrated analysis of medical images
WO2007017500A1 (en) * 2005-08-10 2007-02-15 Sr Labs S.R.L. Method and apparatus for secure insertion of an access code using an eye-tracking device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
KIYOHIKO ABE ET AL: "A system for web browsing by eye-gaze input", ELECTRONICS AND COMMUNICATIONS IN JAPAN, SCRIPTA TECHNICA. NEW YORK, US LNKD- DOI:10.1002/ECJ.10110, vol. 91, no. 5, 1 May 2008 (2008-05-01), pages 11 - 18, XP001523149 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013173148A3 (en) * 2012-05-18 2016-06-09 Microsoft Corporation Interaction and management of devices using gaze detection
US9823742B2 (en) 2012-05-18 2017-11-21 Microsoft Technology Licensing, Llc Interaction and management of devices using gaze detection
US10067563B2 (en) 2012-05-18 2018-09-04 Microsoft Technology Licensing, Llc Interaction and management of devices using gaze detection
ITFI20120165A1 (en) * 2012-08-08 2014-02-09 Sr Labs S R L INTERACTIVE EYE CONTROL MULTIMEDIA SYSTEM FOR ACTIVE AND PASSIVE TRACKING
WO2014024159A1 (en) * 2012-08-08 2014-02-13 Sr Labs S.R.L. Interactive eye-control multimedia system for active and passive tracking

Also Published As

Publication number Publication date
CN102483650A (en) 2012-05-30
DK2476043T3 (en) 2018-03-05
EP2476043B1 (en) 2017-12-06
ES2661554T3 (en) 2018-04-02
US20120173999A1 (en) 2012-07-05
BR112012005342A2 (en) 2021-11-03
EP2476043A1 (en) 2012-07-18
ITFI20090198A1 (en) 2011-03-12
CA2773636A1 (en) 2011-03-17
IT1399456B1 (en) 2013-04-19
US9372605B2 (en) 2016-06-21
PT2476043T (en) 2018-03-16
NO2476043T3 (en) 2018-05-05
CA2773636C (en) 2022-08-02

Similar Documents

Publication Publication Date Title
CA2773636C (en) Method and apparatus for using generic software applications by ocular control and suitable methods of interaction
US9952672B2 (en) Arrangement, method and computer program for controlling a computer apparatus based on eye-tracking
US6963937B1 (en) Method and apparatus for providing configurability and customization of adaptive user-input filtration
EP2266014B1 (en) Apparatus to create, save and format text documents using gaze control and associated method
US8082070B2 (en) Methods and systems for displaying assistance messages to aircraft operators
US20140049462A1 (en) User interface element focus based on user's gaze
US10831346B2 (en) Ergonomic and sensor analysis based user experience design
CN111696546B (en) Using a multimodal interface to facilitate the discovery of verbal commands
Bacher Augmented User Interaction
Istance et al. Eye-based control of standard GUI software
CN114924646A (en) Multi-mode interaction method and system for head-mounted display equipment
Bos Design of an Eye-Gaze Interface for Controlling an Assistive Robot Arm
CN114967918A (en) Multi-mode interaction method and system for head-mounted display equipment
Breuninger Suitability of Touch Gestures and Virtual Physics in Touch Screen User Interfaces for Critical Tasks
Bacher Web2cHMI. A multi-modal native user interface implementation for accelerator operations and maintenance applications
BR112012005342B1 (en) METHOD FOR CONTROLLING THE OPERATION OF AN OPERATING SYSTEM AND ASSOCIATED APPLICATIONS PROGRAM
Rozado et al. RESNA/NCART 2016

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 201080039407.5

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10787877

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 13395014

Country of ref document: US

Ref document number: 2773636

Country of ref document: CA

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2010787877

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 3210/CHENP/2012

Country of ref document: IN

REG Reference to national code

Ref country code: BR

Ref legal event code: B01A

Ref document number: 112012005342

Country of ref document: BR

ENP Entry into the national phase

Ref document number: 112012005342

Country of ref document: BR

Kind code of ref document: A2

Effective date: 20120309

ENP Entry into the national phase

Ref document number: 112012005342

Country of ref document: BR

Kind code of ref document: A2

Effective date: 20120309