WO2022262936A1

WO2022262936A1 - Gaze based method for triggering actions on an operable device

Info

Publication number: WO2022262936A1
Application number: PCT/EP2021/065968
Authority: WO
Inventors: Volodymyr SALO; Frank Linsenmaier; Nils BERGER
Original assignee: Viewpointsystem Gmbh
Priority date: 2021-06-14
Filing date: 2021-06-14
Publication date: 2022-12-22
Also published as: EP4356220A1

Abstract

The present invention concerns a gaze-based method for triggering an action on an operable device provided with a display. The user using a gaze tracking system may command or trigger specific functions on the operable device through predetermined gaze information, movement, fixations.

Description

“GAZE BASED METHOD FOR TRIGGERING ACTIONS ON AN OPERABLE

DEVICE” SPECIFICATION

TECHNICAL BACKGROUND

The present invention discloses a gaze-based method for triggering actions on an operable device, provided with a programmable computer and a display, by a gaze tracking device. Gaze-tracking devices and in particular spectacles usually comprise a sensor, which is oriented onto an eye of the spectacles wearer; the coordinates of the pupil and the viewing direction of the eye can be ascertained using such spectacles and displayed on a correspondent display computer device where a second user is able to watch the gaze direction of the wearer on his relevant field of view, via internet live streaming. Therefore together with a so-called field of vision video, which is prepared by a further field of vision camera arranged on the spectacles in the viewing direction of a user, the point at which the user looks can be ascertained using such spectacles.

As anticipated, these known gaze tracking spectacles are part of an gaze tracking system which may further comprise a computer device able to receive gaze tracking data, such as eye video for pupil detection and field of view video. Said gaze tracking system is able to give, as output, a field of view video, comprising information regarding the user's gaze direction. Said output then is sent to a user operating a display computer device, via internet streaming.

In both the abovementioned configurations the second user is able to see, via internet streaming in real time, what the user wearing the gaze tracking device is looking at in his specific field of view, through a computer device remotely connected to the gaze tracking device.

PRIOR ART

Nowadays are known computer devices which are commanded by the user's voice, in order to play certain songs, answer questions and for entertaining purposes in general. Such devices are also able to activate other operable devices -like robot vacuum cleaner- in turn connected to the same wi-fi network of the above mentioned computer device.

In general concerns regarding these voice commanded devices are related to privacy matters, i.e. users may worry about being registered during personal conversations even when the device is apparently not working. Furthermore another relevant disadvantage of a voice operated device is that, in some conditions, the user's voice does not reach properly the device, thus leading to malfunctions, which shall be corrected manually by the user, in order to trigger correctly the desired action or to receive the desired feedback. In addition a further disadvantage of these voice operated devices is that, in order to trigger a specific action on the operable device, the user shall firstly give a specific command to activate the voice-commanded device to listen the following command, and then pronounce the command-related sentence, to have that specific command/action been performed by the device. This two-step process is very time consuming and not reliable.

Another problem related to voice commanded devices deals with managing simultaneously different users, because of different voice characteristics between young and old persons, man and women and possible command overlapping.

Are also known gaze tracking devices, like gaze tracking spectacles described in US2016/0206196. It discloses eye -tracking spectacles being part of an gaze tracking system comprising a computer device, where runs a software able to implement functionality already mentioned. This prior art document describes eye-tracking spectacles comprising two eye cameras arranged in the nose frame of the glasses and a field of view camera in the middle of the glasses.

In particular according to one preferred embodiment of the prior art invention, it is provided that at least one field of vision camera is arranged on the glasses frame. The field of vision camera is provided to record a field of vision, video including individual and successive field of vision images. The recordings of the two eye acquisition cameras and the at least one field of vision camera can thus be entered in correlation in the field of vision video of the respective gaze point.

Eye/gaze tracking devices may be used for different purposes but are still not used in concrete to allow a specific person to command small household appliances and operable devices in general.

Furthermore in order to authorize and authenticate only a specific person to operate on an operable device like a smartphone are used PIN, passwords, biometric data but passwords are hard to remember and easy to be stolen and biometric authentication, that is identification through a person's distinctive physical or behavioral characteristics, have failed to gain widespread adoption, in part due to usability and also cultural acceptability issues.

Other possibilities are using biometrical data (fingerprint, iris scanning, voice or facial recognition), or choosing the "right" picture from a list, or drawing a pattern upon a mesh with pointing device/a finger. Some of these methods are preferable to use in one environment and not that often used in other ones (s.a. the login/password combination, rarely used on small wearables, due to the necessity to type long strings on the small screen keyboard; such devices usually have only one user, so the login part is omitted and only a short PIN needs to be entered). In other cases are used specific hardware, i.e. the fingerprint device, so used mostly on wearables that have this hardware parts included, and some laptops. The usemame/password combination or the PIN code can be intercepted with the help of spyware, installed on the device by a malicious user or by a virus, along with the image/ pattern can easily be circumvented by a person peeking over the display during the authentication procedure initiated by a user. Also, a fingerprint, facial and voice recognition and iris scanning can be faked, too. So there is a strong need for an authentication method that is at the same time easy to use and hard to circumvent. In addition simple authentication methods using gaze tracking are also known, for instance user interfaces for authenticating people to automated teller machine (ATM). In some cases have been implemented visual stimuluses shown on the ATM screen, while eye tracker capture the user's eye movement data moving through an array of stationary targets that users activated in sequence, using their gaze, while another displayed a passage of text that users are required to read, in order to finalize the authentication process.

In the case explained above, gaze tracking device was used to obtain a consensus from the ATM to withdraw a certain amount of money.

In general all the described authentication method are not secure, because very easy to be faked my a malicious third party observing the correct sequence used by the user. PURPOSES OF THE INVENTION

One objective of the present invention, according to a first of its aspects, is obtaining a method for triggering an action on an operable device provided with a computer device and a display, without any problem related to privacy matters due to accidental voice recording or to unwanted activation, those being typical problems of the known voiced commanded devices.

A second objective of the present invention is providing a method preventing malfunctions of the known voice commanded devices, avoiding the necessity that the device be operated manually by the user to trigger correctly the desired action.

A third objective of the present invention is providing a reliable activation method and a time saving method to trigger a specific action on an operable device.

A fourth objective of the present invention is providing a method usable even in noisy or in silent locations, usable without being noticed by other people or, on the contrary, disturbing other people, or being usable by people having diseases affecting their voice or their speaking skills.

A further objective of the present invention is providing a method for triggering an action on a connected device, by a method easy to be remembered by a user.

Another objective of the present invention is providing a method easy to be used by the user connected via internet to the operable device, avoiding even possible computer device multiuser problems.

A further objective of the present invention is providing a method, securer if compared to known biometrics or gesture methods, when used as a personal authentication method. A final objective of the present invention is providing a method able to solve all mentioned prior art drawbacks disclosed in the present specification.

SUMMARY OF THE INVENTION

Hereinafter are summarized some technical aspects of the present inventions which enable some of the most important purposes to be achieved.

According to a first aspect this invention relates to a gaze-based method for triggering an action on an operable device having a display, said method using a gaze tracking device comprising at least one eye sensor configured to detect pupil position of the wearer and a front camera configured to detect the field of view of said user, the gaze tracking device thus providing gaze information of the user, said action being triggered by a specific user's eye movements and fixation time, on specific positions over a predetermined object/visual area displayed on the operable device.

Such method confers the possibility to avoid any problem related to privacy matters due to accidental voice recording or to unwanted activation of the known voiced commanded devices and a reliable and efficient method, to trigger correctly the desired action on the operable device.

According to a second aspect this invention relates to a method wherein the predetermined position of the at least one object/ visible area related fixation visual point on the relevant visual representations of the user's gaze and the predetermined correspondent fixation time, are parameters uniquely associated to a gaze tracking device when worn by a user, in order to let only the user wearing the gaze tracking device command the operable device.

According to a third aspect this invention relates to a method wherein said action is triggered by a specific user's eye movements and fixation time on at least two specific positions over the predetermined object/ visual area displayed on the operable device. This feature allowing the possibility to implement more than one action triggered by user's gaze and to make the activation process more robust and hard to be replicated by other not allowed users.

According to a fourth aspect this invention relates to a method wherein the display of the operable device can show a correspondent visual feedback, when the user eye movement positions and related fixation times match a predetermined set of positions and related fixation times over said at least one object/visible area, said visual feedback being recognisable by the user.

In this way the user can be aware of the correctness of his/her eye movements fixation time, in order to be able to trigger the desired action. According to a fifth aspect this invention relates to a method wherein visual representation of said visual feedback on the display of the operable device changes depending on the related fixation time.

In this way the user can distinguish correctness of his/her eye movements and related fixation time, in order to be able to trigger the desired action.

According to a sixth aspect this invention relates to a computer readable storage comprising computer-executable instructions which, when executed, configure a processor to perform the method described.

According to a seventh, eighth, nineth aspects this invention relates to an operable device, a gaze tracking device and computer device comprising a processor, a computer readable storage medium coupled to the processor, the computer readable storage medium having stored thereon computer executable instructions which, when executed, configure the processor to perform the corresponding steps of the method described.

According to further aspects this invention relates to further method features claimed in the dependent claims of the present specification.

FIGURES

The structural and functional features of the present invention and its advantages with respect to the known prior art, will become even clearer from the underlying claims, and in particular by an examination of the following description, made with reference to the attached figures which show a preferred but not limited schematic embodiment of the invented method, system, device, in which:

Figure 1 illustrates one preferred embodiment of the system architecture according to the present invention;

Figure 2 illustrates a flow chart of the pairing phase of the method according to the present invention (U=user, G=gaze-tracking device, OD=operable device); Figure 3 illustrates a detailed flow chart of the trigger phase of the method according to the present invention (U=user, G=gaze-tracking device, OD=operable device);

Figure 4 illustrates a simplified flow chart of the method according to the present invention (U=user, G=gaze-tracking device, OD=operable device).

Figure 5 illustrates a possible mesh displayed on the operable device to be used in the method according to the present invention.

Figure 5a illustrates a visual representation of the mesh shown in fig. 5 used in the method according to the present invention.

Figure 6a and 6b illustrate an object with or without gaze movements highlighted possibly displayed on the operable device to be used in the method according to the present invention.

Figure 7 illustrates a schematic representation of eye glance behaviour for fixation.

Figure 8 illustrates a schematic representation of eye glance behaviour with a sequence for initial fixation, a saccade and a second fixation.

DETAILED DESCRIPTION OF THE INVENTION

In general, this disclosure describes a gaze-based method for triggering action on an operable device, provided with a programmable computer and a display, by a gaze tracking device.

The gaze tracking device 1 may have a frame, wherein the frame has at least one receiving opening/lens receptacle opening for a disk-like structure, and wherein the frame has an U-shaped portion where preferably are located a right eye acquisition sensor and a left eye acquisition sensor, said sensors having the purpose of detecting the position of the user's eye, in order to determine continuously his gaze direction when in use.

The frame may have a U-shaped portion provided for arranging the gaze tracking device 1 on the nose of a human. Due to the integration of the eye acquisition sensors in the nose frame parts, the gaze tracking device 1, in a preferred solution, may do not have any protruding parts, and is therefore suitable to be worn as work spectacles or protective spectacles. The viewing movements of a worker or of a human operating a machine can thus not only be monitored and analysed, but rather can be used directly for controlling the machine. In addition, the capability for controlling a machine can thus be monitored, since the physical and mental state of the relevant human can be concluded on the basis of the eye movements, for example, whether he is overtired or under the influence of psychotropic substances. Thus, above all in the case of public use of the gaze tracking device 1, significantly improved results can be achieved, because the measurement accuracy is increased, and also the measurement apparatus can now no longer be perceived by the surroundings and therefore also does not result in negative influences of the test person. The specifications “right” or “left” or “high” or “low” relate to the intended manner of wearing the gaze tracking device 1 by a human being.

As mentioned before, in a preferred solution, the right eye acquisition sensor is arranged in the right nose frame part, and the left eye acquisition sensor is arranged in the left nose frame part of the gaze tracking device. The two eye acquisition sensors may be designed as digital cameras and may have an objective lens. In a preferred solution the two eye acquisition cameras are each provided to observe one eye of the human wearing the relevant gaze tracking device 1, and to prepare in each case an eye video including individual eye images or individual images.

According to one preferred embodiment of gaze tracking device 1, it is provided that at least one field of vision camera is arranged on the gaze tracking device frame, preferably in the U-shaped portion of the frame. The field of vision camera is provided to record a field of vision video, including individual and successive field of vision images. The recordings of the two eye acquisition cameras and the at least one field of vision camera can thus be entered in correlation in the field of vision video of the respective gaze point. A larger number of field of vision cameras can also be arranged in the gaze tracking device 1. For instance the gaze tracking device may be provided with at least two cameras, where at least one of them has a wider Field of View and low resolution and at least one of them with a narrower Field of View but capable of higher resolution.

In order to perform the method according to the present invention it may be used a gaze tracking module, not having the shape of a pair of eye glasses, comprising at least two eye sensors (one for each eye) and a field of view camera as already explained, therefore in any kind of gaze-tracking device.

It is preferably provided that the gaze tracking device 1 have electronical components like a data processing unit and a data interface, the data processing unit may be connected to the right eye acquisition sensor and the left eye acquisition sensor. The gaze tracking device 1 furthermore may have an energy accumulator for the energy supply of the right eye acquisition sensor and the left eye acquisition sensor, and also the data processing unit and the data interface.

According to one particularly preferred embodiment of present gaze tracking device 1 it is provided that the electronical components, including a processor and a connected storage medium, may be arranged in the sideway part of the frame of the gaze tracking device. The entire recording, initial analysis, and storage of the recorded videos can thus be performed in or by the gaze tracking device 1 itself or by a computer device 2 connected to the gaze tracking device 1.

A data processing unit also comprises a data memory. It is preferably designed as a combination of a microcontroller or processor together with a RAM. The data processing unit is connected in a signal-conducting manner to a data interface. It can also be provided that the data interface and the data processing unit are formed jointly in hardware, for example, by an ASIC or an FPGA. The interface is preferably designed as a wireless interface, for example, according to the Bluetooth standard or IEEE 802.x, or as a wired interface, for example, according to the USB standard, wherein in this case the gaze tracking device 1 have a corresponding socket, for example, according to micro- USB. Additional sensors could be inserted in the gaze tracking device 1 and connected with the data processing unit.

The data processing unit and the data interface may be connected at least indirectly to the energy accumulator by circuitry, and are connected in a signal-conducting manner to the field of view camera, the right eye acquisition sensor, and the left eye acquisition sensor.

As already described, the present described method is particularly well suitable for gaze tracking systems comprising a gaze tracking device and a computer device connected to the gaze tracking device via internet, said computer device mounted in general in whatsoever kind of device like a vacuum cleaner robot, a fridge, (household appliances in general), an ATM (automatic teller machine) for instance as shown in fig. 1.

Figures 3 to 4 describe schematically the following steps of the method according to the present invention:

Figure 4 describes a simplified flow chart of the method of the present invention using a gaze tracking device 1, already described, interacting via internet live streaming with an operable device 2 (an ATM, a small household appliance and so on), which is a device comprising at least a programmable computer, a memory and an internet interface enabling it to be always connected with internet, and a display 20, said method comprises the following steps: -100 receiving gaze information identifying movements of the user's gaze on related visual representations 10 of his/her field of view (see fig. 5a) when the user wears the gaze-tracking device 1;

-105 recognize the user via known process like for example a smart card, personal PIN or other known identification method (optional step, not mandatory),

-110 causing a display 20 of the operable device 2 to display an object/ visible area 23, said object/ visible area 23 being visible by the user, said object/ visible area 23 having predetermined geometric and dimensional features and comprising at least one predetermined user-object/visible area-related visual point 24, being distinguishable by the user, previously univocally associated to the user and at least one object/visible area-related visual point 22 being distinguishable by the user;

-having a predetermined fixation time previously univocally associated to the user, said predetermined fixation time being related to the at least one predetermined user- object/visible area-related visual point 24;

-115 while the user is looking at the object/visible area 23, in order to look for the at least one predetermined user-object/visible area-related visual point 24;

-120 identifying possible fixation visual points 21 on the correspondent visual representations 10 of the field of view of the user, based on at least one predetermined fixation criterion;

-130 identifying at least one fixation visual points 21 over said object/visible area 23 visualised in the correspondent visual representations 10, and determining the fixation time of the fixation visual points 21 and the position in relation to the object/visible area 23,

-148 determining if the at least one fixation visual points 21 and its related fixation time on the relevant visual representations 10 match the at least one predetermined user-object/visible area-related visual point 24 and the predetermined correspondent fixation time,

-150 if the at least one fixation visual points 21 and its related fixation time on the relevant visual representations 10 match the at least one predetermined user- object/visible area-related visual point 24 and the predetermined correspondent fixation time, causing to trigger a specific function on that operable device 2, correspondent to the predetermined user-object/visible area-related visual point 24 and to the predetermined correspondent fixation time.

-160 if the at least one fixation visual points 21 and its related fixation time on the relevant visual representations 10 do not match the at least one predetermined user- object/visible area-related visual point 24 and the predetermined correspondent fixation time, then start again from step 115.

It shall be highlighted regarding step 148 that, in order to determine if the at least one fixation visual points 21 and its related fixation time on the relevant visual representations 10 matches the at least one predetermined user-object/ visible area-related visual point 24 and the predetermined correspondent fixation time, it is considered the position of the at least one fixation visual points 21 in correspondence to the object/visible area 23 on the relevant visual representation 10, and then this position is compared to the position of the at least one predetermined user-object/ visible area-related visual point 24 in correspondence of the object/visible area 23, being the at least one predetermined user- object/visible area-related visual point 24 distinguishable and recognizable by the user.

In addition, as already mentioned, the at least one predetermined user-object/visible area- related visual point 24 and the predetermined correspondent fixation time are data previously acquired by the user and univocally associated to the user.

IB If for instance the object/visible area 23 shown on the display 20 of the operable device 2, is made as shown in fig. 5, i.e. the object/ visible area 23 is a 3x3 mesh (comprising 9 possible object/ visible area-related fixation visual points 22), and the user has previously associated the first upper left point of the 3x3 mesh as the at least one predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 2 seconds), then if on the correspondent visual representation 10 (see fig. 5a) it is determined that the at least one fixation visual points 21 matches the first upper left point of the 3x3 mesh, then the action correspondent to said predetermined user-object/ visible area-related visual point 24 is triggered. Therefore in this example the action on the operable device is triggered on the basis of only one predetermined user-object/ visible area-related visual point 24 and its related predetermined fixation time.

The abovementioned set up can be used to trigger simple actions on household appliances, like for instance activating the cleaning mode on a vacuum cleaning robot, or activating a specific function on a fridge.

However, a more complicated sequence of predetermined points might be used to trigger a specific action.

For example (as shown in fig. 5) the user sequence, which has been previously associated to the user, may comprise the first upper left point of the 3x3 mesh as the first predetermined user-object/ visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 2 seconds), then the central point of the 3x3 mesh as the second predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 3 seconds), then the lower right point of the 3x3 mesh as the third predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 1 second), then the lower central point of the 3x3 mesh as the fourth predetermined user- object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 1 second).

Therefore, if on the correspondent visual representations 10 (see fig. 5a) it is determined that first, second, third, fourth fixation visual points 21 match the correspondent first, second, third, fourth predetermined user-object/ visible area-related visual points 24 and their correspondent predetermined fixation times as described above, then the action correspondent to said sequence of four predetermined user-object/ visible area-related visual point 24 is triggered on the operable device 2.

Another example of the object/visible area 23 is shown in fig. 6a. In this case the object/ visible area 23 shown on the display 20 of the operable device 2, is a unicorn, and the user has previously associated the following sequence to a specific action on the operable device 2: the lower point of the front left horse's hoof is the first predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 1 second), the lock of mane to the right of the hom is the second predetermined user-object/ visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 2 second), the lower rightmost point of the rear left horse's hoof is the third predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 3 seconds). Therefore if on the correspondent visual representations 10 (see fig. 5a) it is determined that a first, second, third fixation visual points 21 match the correspondent first, second, third predetermined user-object/visible area-related visual points 24 and their correspondent predetermined fixation times as described above, then the action correspondent to said sequence of three predetermined user-object/ visible area-related visual point 24 is triggered on the operable device 2. Therefore, in this example the action on the operable device 2 is triggered on the basis of at least two predetermined user-object/ visible area-related visual points 24. Number of predetermined user-object/visible area-related visual points 24 needed to trigger an action on the operable device 2 may be decided by the user in case of household appliances, or by the manager of the multiuser operable device 2, in case of highest security level necessities.

Such a complicated sequence of points may be used for implementing an authentication method on an ATM device, to allow only a specific user wearing a determined gaze tracking device 1 to withdrawn cash from the ATM device.

In order to make the authentication method described safer, the sequence path may not be visualised on the display 20 as shown in fig 6b. On the contrary in order to help the user to recognize the entire sequence, the sequence path may be visualised on the display 20 as shown in fig 6a, 5.

In addition another option may be implemented in the method of the present invention: the user is given the possibility to select her/his personal and predetermined image, between a set of images, said predetermined image previously associated univocally to the user.

Figure 3 describes a detailed flow chart of a further preferred solution of the method of the present invention in which at least two predetermined user-object/ visible area-related visual point 24 and the correspondent fixation times are able to cause triggering a specific function on that operable device 2, as in the examples described above, it is highlighted said method may further comprise the following steps:

-134 causing the display 20 to display a correspondent visual feedback 25 when the at least first and second fixation visual points 21 and their related fixation times on the relevant visual representations 10 match the correspondent at least first and second predetermined user-object/visible area-related visual point 24 and the predetermined correspondent fixation times, said visual feedbacks 25 being recognisable by the user. As already explained, if the at least two visual points 21 and their related fixation times on the relevant visual representations 10 match the correspondent at least first and second predetermined user-object/ visible area-related visual point 24 and the predetermined correspondent fixation times, then it is caused to trigger the specific function on that operable device 2 correspondent to the predetermined user-object/visible area-related visual points 24 and to the predetermined correspondent fixation times.

Said visual feedbacks are clearly shown in the examples in fig. 5, 5a, where the first upper left point of the 3x3 mesh is marked with a dark grey circle corresponding to a predetermined fixation time for example of 2 seconds, then the central point of the 3x3 mesh is marked with a black circle corresponding to a predetermined fixation time for example of 3 seconds, and the lower right point of the 3x3 mesh and the lower central point of the 3x3 mesh are marked with a light grey circle corresponding to a predetermined fixation time for example of 1 second.

In figures 5, 5a, 6a said visual feedbacks 25 are visualised over the predetermined user- object/visible area-related visual point 24 but they may be visualised in a totally different area of the display 20, in order to make securer the method. In this way a possible third party is not able to determine the position of the predetermined user-object/visible area- related visual point 24, in view of the position of the correspondent visual feedback 25. A correlation function may be established between the visual feedback 25 and the fixation time, and the visual feedback 25 may change in real time on the display 20, based on different tones of the same colour, or on different dimension of the same shape, or on any visual feedback which could be easily remembered by the user. A user should remember not only the start/ end points of the sequence and all its points, but also the representation of the visual feedback 25 depending on the correspondent fixation time level of each of them.

Furthermore said visual feedback 25 may be implemented also in a method according to the present invention, where the desired action is triggered through only one predetermined user-object/ visible area-related visual point 24 and its related predetermined fixation time.

In a further embodiment said visual feedbacks 25 may be shown in a further device, like a waveguide display connected to the gaze-tracking device 1 and connected via internet live streaming to the operable device 2. In this case the method object of the present invention shall comprise the following step:

-said visual feedback 24 is caused to be displayed on a display provided on the gaze tracking device 1.

In addition according to a further preferred version of the method of the present invention, comprises the following steps:

-138 causing the display 20 to display a correspondent visual feedback 25 when the at least one fixation visual points 21 on the relevant visual representations 10 matches the at least one predetermined user-object/ visible area-related visual point 24, said visual feedback 25 changes depending on the related fixation time of the at least one fixation visual point 21, when the fixation time of the at least one fixation visual point 21 is still not matching the predetermined fixation time related to the predetermined user-object/visible area-related visual point 24;

- the user being able to recognize said visual feedback 25;

—140 having a predetermined visual feedback related to the predetermined fixation time related in turn to the at least one predetermined user-object/ visible area-related visual points 24, previously univocally associated to the user; -142 checking, by the user, if the visual feedback 25 displayed on the display 20 of the operable device 2 in correspondence of at least one fixation visual point 21 corresponds to the predetermined visual feedback related to the correspondent at least one predetermined user-object/visible area-related visual point 24;

- if the visual feedback 25 does not correspond to the predetermined visual feedback, then the user continues to fixate the same at least one fixation visual point 21 on the display 20, in correspondence of the at least one predetermined user-object/visible area-related visual point 24, till the displayed visual feedback 25 corresponds to the predetermined visual feedback related to the at least one predetermined user- object/visible area-related visual point 24;

- if the visual feedback 25 corresponds to the predetermined visual feedback associated to the at least one predetermined user-object/visible area-related visual point 24, then

-144 if one or more predetermined user-object/visible area-related visual points 24 are not yet matched in order to trigger the correspondent action on the operable device 2, then the user goes back to step 130, looking for the following predetermined user-object/visible area-related visual point 24 on said object/visible area 23;

-if all the predetermined user-object/visible area-related visual points 24 are matched, in order to trigger the correspondent action on the operable device 2, then it is caused to trigger the correspondent action on the operable device 2.

As may be understood this sequence combining more than one predetermined user- object/visible area-related visual point 24 associated to related fixation times, is very hard to be got by a criminal, therefore this kind of authentication method is much stronger than its predecessors, as biometrical authentication, without the need to involve additional biometric devices or infrared sensors. As already mentioned, another level of strength can be applied by making the mesh invisible and showing a user-selected image over it, so the mesh points can be matched with the image parts and the user just selects some of them in some predefined order (with predefined fixation time applied to each point), and the resulting pattern is not shown. In this case it's very hard for the interceptor to remember the coloured pattern, and it's almost impossible if the visual feedback fixation time-related is shown only during the pattern application and not shown on resulting pattern afterwards.

When the gaze-tracking device 1 is used, such pattern looks like short eye movements, completely indistinguishable from any other eye movements, so this authentication method is absolutely malicious peek-proof, it is impossible to be recorded and it cannot be faked.

The method of the present invention can be exploited in security-critical environments i.e. banking applications, as well as regular desktop ones for arbitrary use cases.

Still regarding authentication methods, it shall be highlighted that there are two main types of devices, which would benefit from the security advantages, offered by the method: multiuser devices (MUD), i.e. ATM machines, bank safe locks, highly secure access systems, and single-user ones (SUD), i.e. smartphones.

Hereinafter is described the authentication process through an ATM device. The gaze tracking device 1 has a Bluetooth (BLE) adapter, and the operable device 2 (ATM device for example) has one too, therefore they need to paired, so the regular interoperation between gaze tracking device 1 and the operable device 2 could happen. The operable device 2 BLE adapter may always be in pairing mode, while there is a switch (hardware or software) implemented on the operable device 2, which should be triggered to enter the pairing mode; when both devices are in pairing mode and they are in the Bluetooth correct range, they are “visible” to each other. The operable device 2 detects a pairing attempt and starts the authentication method object of the present invention. Then an object/visible area 23 (for example a mesh) is displayed on display 20 and a user is presented with the possibility to authenticate himself/herself to work with the operable device 2. If the user’s authentication was successful, the operable device 2 closes the authentication screen, presents the user with a user interface, and records the user’s gaze tracking device 1 hardware ID to session storage. If the user is unable to authenticate (incorrect attempt), then the gaze tracking device 1 remains paired to the operable device 2. There are several unsuccessful attempts to perform the authentication method on the operable device 2, after which the current pairing is cancelled and the operable device 2 returns to pairing mode. In this case, a user can once more enter the pairing mode on the gaze tracking device 1 to request new authentication. In a further preferred solution the operable device 2 system may record the gaze tracking device 1 hardware ID to the denied session storage to prohibit further login tries for a certain time or indefinitely.

As anticipated, in order to establish a unique connection between the operable device 2 and the gaze-tracking device 1, being the operable device 2 generally a multiuser device, and in order to avoid even possible computer device multiuser problems, preferably the method of the present invention further comprises a pairing phase. The pairing phase is described in the flow chart in fig. 2 and comprises the following steps:

-200 the user sets the gaze tracking device 1 in the pairing mode, while the operable device 2 is already in the pairing mode;

-210 the gaze tracking device 1 enters in the pairing mode

-220 the operable device 2 checks if the gaze-tracking 1 is allowed to be paired with the operable device 2, if not allowed

-224 the gaze-tracking device 1 cancels the pairing attempt and -228 the user sees that the pairing attempt is cancelled If allowed,

-230 the operable device 2 accepts the pairing mode -231 the gaze tracking device 2 is paired -232 the user sees that the pairing attempt is accepted;

-233 the gaze tracking device 2 registers the successful login attempt;

-234 the operable device 2 records the gaze-tracking device 1 hardware ID in the session storage,

-235 the gaze tracking device 1 is successfully connected and activated on the operable device 2

-236 the user sees that the gaze tracking device 1 is successfully connected and activated on the operable device 2.

Then the user operates through her/his gaze-tracking device 1 the operable device 2 using the method hereinbefore described, thus:

240 the gaze tracking device 1 performs certain actions (already described above);

-250 the operable device 2 performs other actions (already described above);

-260 the user sees results of actions performed by the operable device 2;

-270 the user moves away or ends interaction with the operable device 2;

-280 the gaze tracking device 1 unpairs from the operable device 2;

-290 the operable device 2 expunges the gaze-tracking device 1 hardware ID out of the session storage;

-300 the user sees he/ she is logged out of the connection with the operable device 2.

The user’s gaze tracking device 1 may be activated on the operable device 2 only for a relatively small amount of time, i.e. unless the user’s authentication method persists. Whenever the user goes out of the space accessible by the operable device 2 Bluetooth adapter, session storage data associated with it is erased, so next time the whole authentication should be started again. This makes sense since a new user with gaze tracking devicel may enter the accessible area and starts the authentication method during this time. Operable devices 2 are usually devices allowing multiple gaze tracking devices 1 to work with, but not simultaneously. With single user device SUD, meant operable devices 2, it is meant the devices that will be paired >80% time with one gaze tracking device 1.

A single user device SUD, meant operable device 2, pairing and activation method is simpler. A user usually has one gaze tracking device 1 and one or more SUDs, meant operable devices 2 (like household appliances for instance). To start the pairing process, both devices should enter the pairing mode through their Bluetooth adapters. The main difference between the multi user operable device 2 or a single user operable device 2 with the regard to pairing and activation, is that the single user operable device 2 needs to be previously unlocked, ie. the user needs to be already authenticated to use the operable device 2. After both devices are in this mode, the user pairs his/her gaze tracking device 1 on operable device 2, then he/ she can use the newly paired gaze tracking device 1 to perform the authentication method on his single user operable device 2. No deny list is needed since the user owns both devices.

When one gaze tracking device 1 is connected to more than one single user operable devices 2 and/ or many gaze tracking devices 1 are connected to one or more multiuser operable devices 2, Bluetooth multipoint allows the usage of a single peripheral. In fact a user can look at the same time only onto one item, so his/her focal point will be concentrated on one device’s screen, therefore interacting with only one device at a time. The Bluetooth protocol supports specific set of capabilities called profiles for each device, so when you connect two or more devices with the same functional profile, the primary device allows the user to appoint active only one device. Therefore, two (or more) gaze- tracking devices 1 cannot be used with the same multiuser operable device 2, because only one of them will be considered active. In our case, we select the currently activated gaze tracking device 1 as active on the Bluetooth layer. Any other device will wait for activation until the current one gets deactivated by going out of range by the user or simply by logging out.

For the pairing phase could be used the Bluetooth technology or other kind of suitable technology like ANT

The pairing phase may also comprise steps implementing specific visual feedback on the operable device 2 and on the gaze tracking device 1, when the pairing phase is successfully done.

The method described in the present invention may further be implemented in smartphones or any kind of computer device provided with a screen, in particular touchscreen; the method thus being considered a safer authentication method on such devices, replacing known gesture method for example.

The method in the present invention uses important well known fixation concept. One definition of this concept is easily understandable according to figures 7 and 8 and the following paragraphs.

According to figures 7 and 8, directly following example points of vision 37, 38 are at least tested and compared in a comparison device in relation to compliance with at least the first fixation criterion 25. The comparison device can be any suitable device. Preference is given to devices that include electronic logic modules or so-called logic gates, which allow a comparison of input data based on Boolean algorithms. Particular preference is given to devices that use this type of electronic logic modules in integrated form, particularly in the form of processors, microprocessors and/or programmable logic controllers. Particular preference is given to comparison devices that are implemented in a computer.

The comparison device processes so-called visual coordinates, which can be abbreviated in the following as VCO, and which can be determined based on a correlation function described above between a visual field image 79 and an eye image 78, wherein other methods or procedures can be used to determine these VCO.

The first fixation criterion 25 can be any type of criterion, which allows a differentiation between fixations and saccades. The preferred embodiment of the method according to the invention provides that the first fixation criterion 25 is a predefmable first distance 39 around the first point of vision 37, that the first relative distance 44 between the first point of vision 37 and the second point of vision 38 is determined, and that if the first relative distance 44 is less than the first distance 39, the first and second points of vision 37 , 38 are assigned to the first fixation 48, therefore as long as a second point of vision 38 following a first point of vision 37 remains within the foveal area 34 of the first point of vision 37 and thus within the area of ordered perception of the first point of vision 37, ordered perception is not interrupted and thus continues to fulfil the first fixation criterion 25. This is therefore a first fixation 48. A particularly preferred embodiment of the method according to the invention provides that the first distance 39 is a first viewing angle 41, which preferably describes an area 34 assigned to foveal vision, in particular a radius between 0.5° and 1.5°, preferably approximately 1°, and that the distance between the first point of vision 37 and the second point of vision 38 is a first relative angle 42. Based on the visual coordinates determined using a gaze tracking system, it is possible to determine saccades and fixations 48, 49 simply and accurately. FIG. 7 shows a first fixation 48, for example, which is formed from a sequence of four points of vision 37, 38, 69, 70. FIG. 7 also shows the first distance 39, the first viewing angle 41, the first relative distance 44 and the first relative angle 42. Around each of the four points of vision 37, 38, 69, 70 is a first circle 43 with the radius of the first distance 39, wherein it is clearly shown that the following point of vision 38, 69, 70 lies within the first circle 43 with radius first distance 39 of the preceding point of vision 37, 38, 69, and thus the preferred first fixation criteria 25 is met. In order to adapt to objects that are perceived differently or to different people and/ or conditions, a further updated version of the invention provides that the first fixation criterion 25, particularly the first distance 39 and/ or the first viewing angle 41, can be predefined.

FIG. 8 shows a viewing sequence in which not all points of vision 37 , 38 , 69 , 70 , 71,

72 , 73 , 74 , 75 satisfy the first fixation criterion 25. The first four points of vision ( 37, 38 , 69 , 70 satisfy the fixation criterion 25 and together form the first fixation 48, wherein the following three points of vision 71, 72, 73 do not satisfy the first fixation criterion 25. Only the fourth point of vision 74 following the first fixation 28 satisfies the first fixation criterion 25 compared to the third point of vision 73 following the first fixation 48. The third point of vision 73 following the first fixation 48 is therefore the first point of vision

73 of the second fixation 49, which is formed from a total of three points of vision 73, 74, 75. FIGS. 7 and 8 show illustrative examples, although fixations 48, 49 can occur in natural surroundings with a variety of individual points of vision. The area between the last point of vision 70 of the first fixation 48 and the first point of vision 73 of the second fixation 49 forms a saccade, therefore an area without perception. The angle between the last point of vision 70 of the first fixation 48 and the first point of vision 73 of the second fixation 49 is referred to as the first saccade angle 52.

The points of vision 37, 38 assigned to a saccade or a fixation 48 , 49 can now be output for further evaluation, processing or representation. In particular, it can be provided that the first and the second point of vision 37, 38 can be output and marked as the first fixation 48 or the first saccade. The following ones are further fixation and saccade definitions that may be used and implemented in the method to mark a fixation event according to the present invention: -Saccades are rapid movements of the eyes with velocities as high as 500° per second, while in fixations eyes remain relatively still during fixations for about 200—300 ms;

- Fixations are eye movements that stabilize the retina over a stationary object of interest, while Saccades are rapid eye movements used in repositioning the fovea to a new location in the visual environment;

- Using distinction between the periods in which an area of the visual scene is kept on the fovea — a fixation — and periods in which an area of the visual scene is brought onto the fovea — a rapid eye position change called a saccade;

-Defining a saccade when the visual point direction of the gaze tracking device wearer has moved more than a certain angle per time, (i.e. if it has more than a minimal angular velocity). The cutoff criterion in this case may be specified in units of angular velocity.

In the present invention a preferred solution is defining the fixation event criterion occurring when eye movements of the gaze tracking device wearer are stabilized over an object of interest (i.e. matches said object of interest) for a predetermined period of time, preferably in the range 0,5 to 5 sec.

However regarding the condition defined in the step 134, 138, 144, 148, 150, 160 it may be implemented that a boundary area around the fixation visual points 21 on the relevant visual representations 10 in order to check if one fixation visual points 21 on the relevant visual representations 10 matches or not the correspondent one predetermined user- object/visible area-related visual point 24, and possibly the following steps mentioned in this specification, as a safeguard alternative.

The present invention relates furthermore to a gaze-tracking device 1/a computer device connected to the gaze tracking device 1/the operable device 2 comprising a processor, a computer readable storage medium coupled to the processor said computer readable storage medium having stored thereon computer executable instructions which, when executed, configure the processor to perform the corresponding steps of the method already described in the present specification. Object of the present invention is also the computer readable storage medium having stored thereon computer executable instructions which, when executed, configure the processor to perform the corresponding steps of the method already described in the present specification, according to all the embodiments described and disclosed in this specification.

Claims

1. A gaze-based method for triggering an action on an operable device (2), provided with a programmable computer and a display (20), by a gaze tracking device (1) comprising at least one sensor configured to detect pupil position of the user and one camera configured to detect the field of view of the user, thus providing gaze information of said user on the relevant field of view of said user, wherein the method comprises:

(100) receiving via internet live streaming gaze information identifying movements of the user's gaze represented by visual points on related visual representations (10) of his/her field of view, when the gaze tracking device (1) is worn by the user;

(110) causing a display (20) of the operable device (2) to display an object/visible area (23), being visible by the user, said object/visible area (23) having predetermined geometric and dimensional features and comprising at least one predetermined user- object/visible area-related visual point (24) being distinguishable by the user, and at least one object/visible area-related visual point (22) being distinguishable by the user; (120) identifying possible fixation visual points (21) on the correspondent visual representations (10) of the field of view of the user, based on at least one predetermined fixation criterion related to a predetermined fixation time;

(130) identifying at least one fixation visual points (21) on said object/visible area (23) when said object/visible area is visualised in the correspondent visual representations (10), and determining the fixation time of the one fixation visual points (21) and the position in relation to the object/visible area of said one fixation visual points (21), when said object/visible area is visualised in the correspondent visual representations (10),

(148) having a predetermined user-object/ visible area-related visual point (24) and a correspondent predetermined fixation time and determining if the at least one fixation visual points (21) and its related fixation time on the relevant visual representations (10) match the at least one predetermined user-object/visible area-related visual point (24) and the predetermined correspondent fixation time,

(150) if the at least one fixation visual points (21) and its related fixation time on the relevant visual representations (10) match the at least one predetermined user- object/visible area-related visual point (24) and the predetermined correspondent fixation time, causing to trigger a specific function on that operable device (2) correspondent to the predetermined user-object/visible area-related visual point (24) and to the predetermined correspondent fixation time.

2. The method according to claim 1 wherein the predetermined user-object/visible area- related visual point (24) and the correspondent predetermined fixation time are parameters uniquely associated to an individual user.

3. The method according to any of the preceding claims 1 to 2 wherein only a combination of at least two predetermined user-object/ visible area-related visual point (24) and the correspondent predetermined fixation times is able to cause triggering a specific function on that operable device (2) when the at least two fixation visual points (21) and their related fixation times on the relevant visual representations (10) match the at least two predetermined user-object/ visible area-related visual points (24) and the predetermined correspondent fixation times, thus being able to create different combinations associated to different action triggerable on the operable device (2).

4. The method according to claims 2 or 3 wherein the object/ visible area (23) is a mesh comprising a predetermined number of possible gaze position placed in a precise geometric order or an image.

5. The method according to any of the preceding claims further comprising the following steps:

BO (134) causing the display (20) displaying a correspondent visual feedback (25) when at least one fixation visual points (21) and its related fixation time on the relevant visual representations (10) match the at least one predetermined user-object/ visible area- related visual point (24) and the predetermined correspondent fixation time, said visual feedback (25) being recognisable by the user;

6. The method according to any of the preceding claims 1 to 4 further comprising the following steps:

(138) causing the display (20) to display a correspondent visual feedback (25) when the at least one fixation visual points 21 on the relevant visual representations (10) matches the at least one predetermined user-object/ visible area-related visual point (24), said visual feedback (25) changes depending on the related fixation time of the at least one fixation visual point (21), when the fixation time of the at least one fixation visual point (21) is still not matching the predetermined fixation time related to the predetermined user-object/visible area-related visual point (24).

7. The method according to claim 5 or 6 wherein the visual representation of said visual feedback (25) on the display (20) of the operable device (2) changes depending on the related fixation time.

8. The method according to claim 7 wherein said change in the visual representation of the visual feedback (25) on the display (20) of the operable device (2) is related to its colour and/ or dimension depending on the related fixation time.

9. The method according to claim 7 or 8 wherein said change in the visual representation of the visual feedback (25) on the display (20) of the operable device (2) is preliminary associated to the user.

10. The method according to claim 5 or 6 wherein said visual feedback (25) is caused to be displayed on a display provided on the gaze tracking device (1).

11. An authentication method to authenticate a user onto a operable device (2) using the method according to any of the preceding claims 1 to 10.

12. A computer readable storage medium comprising computer-executable instructions which, when executed, configure a processor to perform the method of any of claims 1 to 11.

13. An operable device (2) comprising a processor, a computer readable storage medium coupled to the processor said memory according to claim 12, the computer readable storage medium having stored thereon computer executable instructions which, when executed, configure the processor to perform the corresponding steps of the method of any of claims 1 to 11.

14. A gaze-tracking device (1) comprising a processor, a computer readable storage medium coupled to the processor said computer readable storage medium according to claim 12, the computer readable storage medium having stored thereon computer executable instructions which, when executed, configure the processor to perform the corresponding steps of the method of any of claims 1 to 11.

15. A computer device connectable to a gaze-tracking device (1) comprising a processor, a computer readable storage medium coupled to the processor said computer readable storage medium according to claim 12, the computer readable storage medium having stored thereon computer executable instructions which, when executed, configure the processor to perform the corresponding steps of the method of any of claims 1 to 11.