EP4356220A1 - Procédé basé sur le regard pour déclencher des actions sur un dispositif actionnable - Google Patents

Procédé basé sur le regard pour déclencher des actions sur un dispositif actionnable

Info

Publication number
EP4356220A1
EP4356220A1 EP21733769.0A EP21733769A EP4356220A1 EP 4356220 A1 EP4356220 A1 EP 4356220A1 EP 21733769 A EP21733769 A EP 21733769A EP 4356220 A1 EP4356220 A1 EP 4356220A1
Authority
EP
European Patent Office
Prior art keywords
visual
user
fixation
predetermined
visible area
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP21733769.0A
Other languages
German (de)
English (en)
Inventor
Volodymyr SALO
Frank Linsenmaier
Nils BERGER
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Viewpointsystem GmbH
Original Assignee
Viewpointsystem GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Viewpointsystem GmbH filed Critical Viewpointsystem GmbH
Publication of EP4356220A1 publication Critical patent/EP4356220A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements

Definitions

  • the present invention discloses a gaze-based method for triggering actions on an operable device, provided with a programmable computer and a display, by a gaze tracking device.
  • Gaze-tracking devices and in particular spectacles usually comprise a sensor, which is oriented onto an eye of the spectacles wearer; the coordinates of the pupil and the viewing direction of the eye can be ascertained using such spectacles and displayed on a correspondent display computer device where a second user is able to watch the gaze direction of the wearer on his relevant field of view, via internet live streaming. Therefore together with a so-called field of vision video, which is prepared by a further field of vision camera arranged on the spectacles in the viewing direction of a user, the point at which the user looks can be ascertained using such spectacles.
  • these known gaze tracking spectacles are part of an gaze tracking system which may further comprise a computer device able to receive gaze tracking data, such as eye video for pupil detection and field of view video. Said gaze tracking system is able to give, as output, a field of view video, comprising information regarding the user's gaze direction. Said output then is sent to a user operating a display computer device, via internet streaming.
  • a computer device able to receive gaze tracking data, such as eye video for pupil detection and field of view video.
  • Said gaze tracking system is able to give, as output, a field of view video, comprising information regarding the user's gaze direction. Said output then is sent to a user operating a display computer device, via internet streaming.
  • the second user is able to see, via internet streaming in real time, what the user wearing the gaze tracking device is looking at in his specific field of view, through a computer device remotely connected to the gaze tracking device.
  • voice commanded devices In general concerns regarding these voice commanded devices are related to privacy matters, i.e. users may worry about being registered during personal conversations even when the device is apparently not working. Furthermore another relevant disadvantage of a voice operated device is that, in some conditions, the user's voice does not reach properly the device, thus leading to malfunctions, which shall be corrected manually by the user, in order to trigger correctly the desired action or to receive the desired feedback. In addition a further disadvantage of these voice operated devices is that, in order to trigger a specific action on the operable device, the user shall firstly give a specific command to activate the voice-commanded device to listen the following command, and then pronounce the command-related sentence, to have that specific command/action been performed by the device. This two-step process is very time consuming and not reliable.
  • Another problem related to voice commanded devices deals with managing simultaneously different users, because of different voice characteristics between young and old persons, man and women and possible command overlapping.
  • Eye -tracking spectacles being part of an gaze tracking system comprising a computer device, where runs a software able to implement functionality already mentioned.
  • This prior art document describes eye-tracking spectacles comprising two eye cameras arranged in the nose frame of the glasses and a field of view camera in the middle of the glasses.
  • At least one field of vision camera is arranged on the glasses frame.
  • the field of vision camera is provided to record a field of vision, video including individual and successive field of vision images.
  • the recordings of the two eye acquisition cameras and the at least one field of vision camera can thus be entered in correlation in the field of vision video of the respective gaze point.
  • Eye/gaze tracking devices may be used for different purposes but are still not used in concrete to allow a specific person to command small household appliances and operable devices in general.
  • biometrical data fingerprint, iris scanning, voice or facial recognition
  • choosing the "right" picture from a list, or drawing a pattern upon a mesh with pointing device/a finger Some of these methods are preferable to use in one environment and not that often used in other ones (s.a. the login/password combination, rarely used on small wearables, due to the necessity to type long strings on the small screen keyboard; such devices usually have only one user, so the login part is omitted and only a short PIN needs to be entered).
  • specific hardware i.e. the fingerprint device, so used mostly on wearables that have this hardware parts included, and some laptops.
  • the usemame/password combination or the PIN code can be intercepted with the help of spyware, installed on the device by a malicious user or by a virus, along with the image/ pattern can easily be circumvented by a person peeking over the display during the authentication procedure initiated by a user. Also, a fingerprint, facial and voice recognition and iris scanning can be faked, too. So there is a strong need for an authentication method that is at the same time easy to use and hard to circumvent. In addition simple authentication methods using gaze tracking are also known, for instance user interfaces for authenticating people to automated teller machine (ATM).
  • ATM automated teller machine
  • eye tracker capture the user's eye movement data moving through an array of stationary targets that users activated in sequence, using their gaze, while another displayed a passage of text that users are required to read, in order to finalize the authentication process.
  • gaze tracking device was used to obtain a consensus from the ATM to withdraw a certain amount of money.
  • One objective of the present invention is obtaining a method for triggering an action on an operable device provided with a computer device and a display, without any problem related to privacy matters due to accidental voice recording or to unwanted activation, those being typical problems of the known voiced commanded devices.
  • a second objective of the present invention is providing a method preventing malfunctions of the known voice commanded devices, avoiding the necessity that the device be operated manually by the user to trigger correctly the desired action.
  • a third objective of the present invention is providing a reliable activation method and a time saving method to trigger a specific action on an operable device.
  • a fourth objective of the present invention is providing a method usable even in noisy or in silent locations, usable without being noticed by other people or, on the contrary, disturbing other people, or being usable by people having diseases affecting their voice or their speaking skills.
  • a further objective of the present invention is providing a method for triggering an action on a connected device, by a method easy to be remembered by a user.
  • Another objective of the present invention is providing a method easy to be used by the user connected via internet to the operable device, avoiding even possible computer device multiuser problems.
  • a further objective of the present invention is providing a method, securer if compared to known biometrics or gesture methods, when used as a personal authentication method.
  • a final objective of the present invention is providing a method able to solve all mentioned prior art drawbacks disclosed in the present specification.
  • this invention relates to a gaze-based method for triggering an action on an operable device having a display, said method using a gaze tracking device comprising at least one eye sensor configured to detect pupil position of the wearer and a front camera configured to detect the field of view of said user, the gaze tracking device thus providing gaze information of the user, said action being triggered by a specific user's eye movements and fixation time, on specific positions over a predetermined object/visual area displayed on the operable device.
  • Such method confers the possibility to avoid any problem related to privacy matters due to accidental voice recording or to unwanted activation of the known voiced commanded devices and a reliable and efficient method, to trigger correctly the desired action on the operable device.
  • this invention relates to a method wherein the predetermined position of the at least one object/ visible area related fixation visual point on the relevant visual representations of the user's gaze and the predetermined correspondent fixation time, are parameters uniquely associated to a gaze tracking device when worn by a user, in order to let only the user wearing the gaze tracking device command the operable device.
  • this invention relates to a method wherein said action is triggered by a specific user's eye movements and fixation time on at least two specific positions over the predetermined object/ visual area displayed on the operable device.
  • this invention relates to a method wherein the display of the operable device can show a correspondent visual feedback, when the user eye movement positions and related fixation times match a predetermined set of positions and related fixation times over said at least one object/visible area, said visual feedback being recognisable by the user.
  • this invention relates to a method wherein visual representation of said visual feedback on the display of the operable device changes depending on the related fixation time.
  • this invention relates to a computer readable storage comprising computer-executable instructions which, when executed, configure a processor to perform the method described.
  • this invention relates to an operable device, a gaze tracking device and computer device comprising a processor, a computer readable storage medium coupled to the processor, the computer readable storage medium having stored thereon computer executable instructions which, when executed, configure the processor to perform the corresponding steps of the method described.
  • Figure 1 illustrates one preferred embodiment of the system architecture according to the present invention
  • Figure 5 illustrates a possible mesh displayed on the operable device to be used in the method according to the present invention.
  • Figure 5a illustrates a visual representation of the mesh shown in fig. 5 used in the method according to the present invention.
  • Figure 6a and 6b illustrate an object with or without gaze movements highlighted possibly displayed on the operable device to be used in the method according to the present invention.
  • Figure 7 illustrates a schematic representation of eye glance behaviour for fixation.
  • Figure 8 illustrates a schematic representation of eye glance behaviour with a sequence for initial fixation, a saccade and a second fixation.
  • this disclosure describes a gaze-based method for triggering action on an operable device, provided with a programmable computer and a display, by a gaze tracking device.
  • the gaze tracking device 1 may have a frame, wherein the frame has at least one receiving opening/lens receptacle opening for a disk-like structure, and wherein the frame has an U-shaped portion where preferably are located a right eye acquisition sensor and a left eye acquisition sensor, said sensors having the purpose of detecting the position of the user's eye, in order to determine continuously his gaze direction when in use.
  • the frame may have a U-shaped portion provided for arranging the gaze tracking device 1 on the nose of a human. Due to the integration of the eye acquisition sensors in the nose frame parts, the gaze tracking device 1, in a preferred solution, may do not have any protruding parts, and is therefore suitable to be worn as work spectacles or protective spectacles.
  • the viewing movements of a worker or of a human operating a machine can thus not only be monitored and analysed, but rather can be used directly for controlling the machine.
  • the capability for controlling a machine can thus be monitored, since the physical and mental state of the relevant human can be concluded on the basis of the eye movements, for example, whether he is overtired or under the influence of psychotropic substances.
  • the right eye acquisition sensor is arranged in the right nose frame part
  • the left eye acquisition sensor is arranged in the left nose frame part of the gaze tracking device.
  • the two eye acquisition sensors may be designed as digital cameras and may have an objective lens.
  • the two eye acquisition cameras are each provided to observe one eye of the human wearing the relevant gaze tracking device 1, and to prepare in each case an eye video including individual eye images or individual images.
  • At least one field of vision camera is arranged on the gaze tracking device frame, preferably in the U-shaped portion of the frame.
  • the field of vision camera is provided to record a field of vision video, including individual and successive field of vision images.
  • the recordings of the two eye acquisition cameras and the at least one field of vision camera can thus be entered in correlation in the field of vision video of the respective gaze point.
  • a larger number of field of vision cameras can also be arranged in the gaze tracking device 1.
  • the gaze tracking device may be provided with at least two cameras, where at least one of them has a wider Field of View and low resolution and at least one of them with a narrower Field of View but capable of higher resolution.
  • a gaze tracking module not having the shape of a pair of eye glasses, comprising at least two eye sensors (one for each eye) and a field of view camera as already explained, therefore in any kind of gaze-tracking device.
  • the gaze tracking device 1 have electronical components like a data processing unit and a data interface, the data processing unit may be connected to the right eye acquisition sensor and the left eye acquisition sensor.
  • the gaze tracking device 1 furthermore may have an energy accumulator for the energy supply of the right eye acquisition sensor and the left eye acquisition sensor, and also the data processing unit and the data interface.
  • the electronical components including a processor and a connected storage medium, may be arranged in the sideway part of the frame of the gaze tracking device.
  • the entire recording, initial analysis, and storage of the recorded videos can thus be performed in or by the gaze tracking device 1 itself or by a computer device 2 connected to the gaze tracking device 1.
  • a data processing unit also comprises a data memory. It is preferably designed as a combination of a microcontroller or processor together with a RAM.
  • the data processing unit is connected in a signal-conducting manner to a data interface. It can also be provided that the data interface and the data processing unit are formed jointly in hardware, for example, by an ASIC or an FPGA.
  • the interface is preferably designed as a wireless interface, for example, according to the Bluetooth standard or IEEE 802.x, or as a wired interface, for example, according to the USB standard, wherein in this case the gaze tracking device 1 have a corresponding socket, for example, according to micro- USB. Additional sensors could be inserted in the gaze tracking device 1 and connected with the data processing unit.
  • the data processing unit and the data interface may be connected at least indirectly to the energy accumulator by circuitry, and are connected in a signal-conducting manner to the field of view camera, the right eye acquisition sensor, and the left eye acquisition sensor.
  • the present described method is particularly well suitable for gaze tracking systems
  • a gaze tracking device and a computer device connected to the gaze tracking device via internet
  • said computer device mounted in general in whatsoever kind of device like a vacuum cleaner robot, a fridge, (household appliances in general), an ATM (automatic teller machine) for instance as shown in fig. 1.
  • FIGS 3 to 4 describe schematically the following steps of the method according to the present invention:
  • FIG. 4 describes a simplified flow chart of the method of the present invention using a gaze tracking device 1, already described, interacting via internet live streaming with an operable device 2 (an ATM, a small household appliance and so on), which is a device comprising at least a programmable computer, a memory and an internet interface enabling it to be always connected with internet, and a display 20, said method comprises the following steps: -100 receiving gaze information identifying movements of the user's gaze on related visual representations 10 of his/her field of view (see fig. 5a) when the user wears the gaze-tracking device 1;
  • -105 recognize the user via known process like for example a smart card, personal PIN or other known identification method (optional step, not mandatory),
  • a display 20 of the operable device 2 to display an object/ visible area 23, said object/ visible area 23 being visible by the user, said object/ visible area 23 having predetermined geometric and dimensional features and comprising at least one predetermined user-object/visible area-related visual point 24, being distinguishable by the user, previously univocally associated to the user and at least one object/visible area-related visual point 22 being distinguishable by the user;
  • predetermined fixation time previously univocally associated to the user, said predetermined fixation time being related to the at least one predetermined user- object/visible area-related visual point 24;
  • the at least one fixation visual points 21 and its related fixation time on the relevant visual representations 10 match the at least one predetermined user- object/visible area-related visual point 24 and the predetermined correspondent fixation time, causing to trigger a specific function on that operable device 2, correspondent to the predetermined user-object/visible area-related visual point 24 and to the predetermined correspondent fixation time.
  • step 115 if the at least one fixation visual points 21 and its related fixation time on the relevant visual representations 10 do not match the at least one predetermined user- object/visible area-related visual point 24 and the predetermined correspondent fixation time, then start again from step 115.
  • step 148 in order to determine if the at least one fixation visual points 21 and its related fixation time on the relevant visual representations 10 matches the at least one predetermined user-object/ visible area-related visual point 24 and the predetermined correspondent fixation time, it is considered the position of the at least one fixation visual points 21 in correspondence to the object/visible area 23 on the relevant visual representation 10, and then this position is compared to the position of the at least one predetermined user-object/ visible area-related visual point 24 in correspondence of the object/visible area 23, being the at least one predetermined user- object/visible area-related visual point 24 distinguishable and recognizable by the user.
  • the at least one predetermined user-object/visible area- related visual point 24 and the predetermined correspondent fixation time are data previously acquired by the user and univocally associated to the user.
  • the object/visible area 23 shown on the display 20 of the operable device 2 is made as shown in fig. 5, i.e. the object/ visible area 23 is a 3x3 mesh (comprising 9 possible object/ visible area-related fixation visual points 22), and the user has previously associated the first upper left point of the 3x3 mesh as the at least one predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 2 seconds), then if on the correspondent visual representation 10 (see fig. 5a) it is determined that the at least one fixation visual points 21 matches the first upper left point of the 3x3 mesh, then the action correspondent to said predetermined user-object/ visible area-related visual point 24 is triggered. Therefore in this example the action on the operable device is triggered on the basis of only one predetermined user-object/ visible area-related visual point 24 and its related predetermined fixation time.
  • the abovementioned set up can be used to trigger simple actions on household appliances, like for instance activating the cleaning mode on a vacuum cleaning robot, or activating a specific function on a fridge.
  • the user sequence which has been previously associated to the user, may comprise the first upper left point of the 3x3 mesh as the first predetermined user-object/ visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 2 seconds), then the central point of the 3x3 mesh as the second predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 3 seconds), then the lower right point of the 3x3 mesh as the third predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 1 second), then the lower central point of the 3x3 mesh as the fourth predetermined user- object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 1 second).
  • a certain predetermined fixation time for example 2 seconds
  • the central point of the 3x3 mesh as the second predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 3 seconds)
  • first, second, third, fourth fixation visual points 21 match the correspondent first, second, third, fourth predetermined user-object/ visible area-related visual points 24 and their correspondent predetermined fixation times as described above, then the action correspondent to said sequence of four predetermined user-object/ visible area-related visual point 24 is triggered on the operable device 2.
  • the object/visible area 23 shown in fig. 6a is shown in fig. 6a.
  • the object/ visible area 23 shown on the display 20 of the operable device 2 is a unicorn, and the user has previously associated the following sequence to a specific action on the operable device 2: the lower point of the front left horse's hoof is the first predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 1 second), the lock of mane to the right of the hom is the second predetermined user-object/ visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 2 second), the lower rightmost point of the rear left horse's hoof is the third predetermined user-object/visible area-related visual point 24 correlated to a certain predetermined fixation time (for example 3 seconds).
  • a certain predetermined fixation time for example 1 second
  • the lock of mane to the right of the hom is the second predetermined user-object/ visible area-related visual point 24 correlated to a certain
  • the action correspondent to said sequence of three predetermined user-object/ visible area-related visual point 24 is triggered on the operable device 2. Therefore, in this example the action on the operable device 2 is triggered on the basis of at least two predetermined user-object/ visible area-related visual points 24. Number of predetermined user-object/visible area-related visual points 24 needed to trigger an action on the operable device 2 may be decided by the user in case of household appliances, or by the manager of the multiuser operable device 2, in case of highest security level necessities.
  • Such a complicated sequence of points may be used for implementing an authentication method on an ATM device, to allow only a specific user wearing a determined gaze tracking device 1 to withdrawn cash from the ATM device.
  • the sequence path may not be visualised on the display 20 as shown in fig 6b.
  • the sequence path may be visualised on the display 20 as shown in fig 6a, 5.
  • the user is given the possibility to select her/his personal and predetermined image, between a set of images, said predetermined image previously associated univocally to the user.
  • Figure 3 describes a detailed flow chart of a further preferred solution of the method of the present invention in which at least two predetermined user-object/ visible area-related visual point 24 and the correspondent fixation times are able to cause triggering a specific function on that operable device 2, as in the examples described above, it is highlighted said method may further comprise the following steps:
  • the display 20 to display a correspondent visual feedback 25 when the at least first and second fixation visual points 21 and their related fixation times on the relevant visual representations 10 match the correspondent at least first and second predetermined user-object/visible area-related visual point 24 and the predetermined correspondent fixation times, said visual feedbacks 25 being recognisable by the user.
  • the at least two visual points 21 and their related fixation times on the relevant visual representations 10 match the correspondent at least first and second predetermined user-object/ visible area-related visual point 24 and the predetermined correspondent fixation times, then it is caused to trigger the specific function on that operable device 2 correspondent to the predetermined user-object/visible area-related visual points 24 and to the predetermined correspondent fixation times.
  • said visual feedbacks 25 are visualised over the predetermined user- object/visible area-related visual point 24 but they may be visualised in a totally different area of the display 20, in order to make securer the method. In this way a possible third party is not able to determine the position of the predetermined user-object/visible area- related visual point 24, in view of the position of the correspondent visual feedback 25.
  • a correlation function may be established between the visual feedback 25 and the fixation time, and the visual feedback 25 may change in real time on the display 20, based on different tones of the same colour, or on different dimension of the same shape, or on any visual feedback which could be easily remembered by the user.
  • a user should remember not only the start/ end points of the sequence and all its points, but also the representation of the visual feedback 25 depending on the correspondent fixation time level of each of them.
  • said visual feedback 25 may be implemented also in a method according to the present invention, where the desired action is triggered through only one predetermined user-object/ visible area-related visual point 24 and its related predetermined fixation time.
  • said visual feedbacks 25 may be shown in a further device, like a waveguide display connected to the gaze-tracking device 1 and connected via internet live streaming to the operable device 2.
  • the method object of the present invention shall comprise the following step:
  • -said visual feedback 24 is caused to be displayed on a display provided on the gaze tracking device 1.
  • the display 20 to display a correspondent visual feedback 25 when the at least one fixation visual points 21 on the relevant visual representations 10 matches the at least one predetermined user-object/ visible area-related visual point 24, said visual feedback 25 changes depending on the related fixation time of the at least one fixation visual point 21, when the fixation time of the at least one fixation visual point 21 is still not matching the predetermined fixation time related to the predetermined user-object/visible area-related visual point 24;
  • —140 having a predetermined visual feedback related to the predetermined fixation time related in turn to the at least one predetermined user-object/ visible area-related visual points 24, previously univocally associated to the user; -142 checking, by the user, if the visual feedback 25 displayed on the display 20 of the operable device 2 in correspondence of at least one fixation visual point 21 corresponds to the predetermined visual feedback related to the correspondent at least one predetermined user-object/visible area-related visual point 24;
  • the user continues to fixate the same at least one fixation visual point 21 on the display 20, in correspondence of the at least one predetermined user-object/visible area-related visual point 24, till the displayed visual feedback 25 corresponds to the predetermined visual feedback related to the at least one predetermined user- object/visible area-related visual point 24;
  • step 130 looking for the following predetermined user-object/visible area-related visual point 24 on said object/visible area 23;
  • this sequence combining more than one predetermined user- object/visible area-related visual point 24 associated to related fixation times is very hard to be got by a criminal, therefore this kind of authentication method is much stronger than its predecessors, as biometrical authentication, without the need to involve additional biometric devices or infrared sensors.
  • another level of strength can be applied by making the mesh invisible and showing a user-selected image over it, so the mesh points can be matched with the image parts and the user just selects some of them in some predefined order (with predefined fixation time applied to each point), and the resulting pattern is not shown. In this case it's very hard for the interceptor to remember the coloured pattern, and it's almost impossible if the visual feedback fixation time-related is shown only during the pattern application and not shown on resulting pattern afterwards.
  • the method of the present invention can be exploited in security-critical environments i.e. banking applications, as well as regular desktop ones for arbitrary use cases.
  • multiuser devices i.e. ATM machines, bank safe locks, highly secure access systems, and single-user ones (SUD), i.e. smartphones.
  • SMD single-user ones
  • the gaze tracking device 1 has a Bluetooth (BLE) adapter, and the operable device 2 (ATM device for example) has one too, therefore they need to paired, so the regular interoperation between gaze tracking device 1 and the operable device 2 could happen.
  • the operable device 2 BLE adapter may always be in pairing mode, while there is a switch (hardware or software) implemented on the operable device 2, which should be triggered to enter the pairing mode; when both devices are in pairing mode and they are in the Bluetooth correct range, they are “visible” to each other.
  • the operable device 2 detects a pairing attempt and starts the authentication method object of the present invention.
  • an object/visible area 23 (for example a mesh) is displayed on display 20 and a user is presented with the possibility to authenticate himself/herself to work with the operable device 2. If the user’s authentication was successful, the operable device 2 closes the authentication screen, presents the user with a user interface, and records the user’s gaze tracking device 1 hardware ID to session storage. If the user is unable to authenticate (incorrect attempt), then the gaze tracking device 1 remains paired to the operable device 2. There are several unsuccessful attempts to perform the authentication method on the operable device 2, after which the current pairing is cancelled and the operable device 2 returns to pairing mode. In this case, a user can once more enter the pairing mode on the gaze tracking device 1 to request new authentication. In a further preferred solution the operable device 2 system may record the gaze tracking device 1 hardware ID to the denied session storage to prohibit further login tries for a certain time or indefinitely.
  • the operable device 2 system may record the gaze tracking device 1 hardware ID to the denied session storage to prohibit further login tries for a certain time or indefinitely
  • the method of the present invention further comprises a pairing phase.
  • the pairing phase is described in the flow chart in fig. 2 and comprises the following steps:
  • the user sets the gaze tracking device 1 in the pairing mode, while the operable device 2 is already in the pairing mode;
  • the gaze tracking device 1 enters in the pairing mode
  • the operable device 2 checks if the gaze-tracking 1 is allowed to be paired with the operable device 2, if not allowed
  • the gaze-tracking device 1 cancels the pairing attempt and -228 the user sees that the pairing attempt is cancelled If allowed,
  • the operable device 2 accepts the pairing mode -231 the gaze tracking device 2 is paired -232 the user sees that the pairing attempt is accepted;
  • the gaze tracking device 2 registers the successful login attempt
  • the operable device 2 records the gaze-tracking device 1 hardware ID in the session storage
  • the gaze tracking device 1 is successfully connected and activated on the operable device 2
  • the gaze tracking device 1 performs certain actions (already described above);
  • the gaze tracking device 1 unpairs from the operable device 2;
  • the operable device 2 expunges the gaze-tracking device 1 hardware ID out of the session storage;
  • the user’s gaze tracking device 1 may be activated on the operable device 2 only for a relatively small amount of time, i.e. unless the user’s authentication method persists. Whenever the user goes out of the space accessible by the operable device 2 Bluetooth adapter, session storage data associated with it is erased, so next time the whole authentication should be started again. This makes sense since a new user with gaze tracking devicel may enter the accessible area and starts the authentication method during this time.
  • Operable devices 2 are usually devices allowing multiple gaze tracking devices 1 to work with, but not simultaneously. With single user device SUD, meant operable devices 2, it is meant the devices that will be paired >80% time with one gaze tracking device 1.
  • a single user device SUD meant operable device 2
  • pairing and activation method is simpler.
  • a user usually has one gaze tracking device 1 and one or more SUDs, meant operable devices 2 (like household appliances for instance).
  • both devices should enter the pairing mode through their Bluetooth adapters.
  • the main difference between the multi user operable device 2 or a single user operable device 2 with the regard to pairing and activation, is that the single user operable device 2 needs to be previously unlocked, ie. the user needs to be already authenticated to use the operable device 2.
  • the user pairs his/her gaze tracking device 1 on operable device 2, then he/ she can use the newly paired gaze tracking device 1 to perform the authentication method on his single user operable device 2. No deny list is needed since the user owns both devices.
  • Bluetooth multipoint allows the usage of a single peripheral. In fact a user can look at the same time only onto one item, so his/her focal point will be concentrated on one device’s screen, therefore interacting with only one device at a time.
  • the Bluetooth protocol supports specific set of capabilities called profiles for each device, so when you connect two or more devices with the same functional profile, the primary device allows the user to appoint active only one device. Therefore, two (or more) gaze- tracking devices 1 cannot be used with the same multiuser operable device 2, because only one of them will be considered active. In our case, we select the currently activated gaze tracking device 1 as active on the Bluetooth layer. Any other device will wait for activation until the current one gets deactivated by going out of range by the user or simply by logging out.
  • the pairing phase could be used the Bluetooth technology or other kind of suitable technology like ANT
  • the pairing phase may also comprise steps implementing specific visual feedback on the operable device 2 and on the gaze tracking device 1, when the pairing phase is successfully done.
  • the method described in the present invention may further be implemented in smartphones or any kind of computer device provided with a screen, in particular touchscreen; the method thus being considered a safer authentication method on such devices, replacing known gesture method for example.
  • the method in the present invention uses important well known fixation concept.
  • One definition of this concept is easily understandable according to figures 7 and 8 and the following paragraphs.
  • the comparison device can be any suitable device. Preference is given to devices that include electronic logic modules or so-called logic gates, which allow a comparison of input data based on Boolean algorithms. Particular preference is given to devices that use this type of electronic logic modules in integrated form, particularly in the form of processors, microprocessors and/or programmable logic controllers. Particular preference is given to comparison devices that are implemented in a computer.
  • the comparison device processes so-called visual coordinates, which can be abbreviated in the following as VCO, and which can be determined based on a correlation function described above between a visual field image 79 and an eye image 78, wherein other methods or procedures can be used to determine these VCO.
  • VCO visual coordinates
  • the first fixation criterion 25 can be any type of criterion, which allows a differentiation between fixations and saccades.
  • the preferred embodiment of the method according to the invention provides that the first fixation criterion 25 is a predefmable first distance 39 around the first point of vision 37, that the first relative distance 44 between the first point of vision 37 and the second point of vision 38 is determined, and that if the first relative distance 44 is less than the first distance 39, the first and second points of vision 37 , 38 are assigned to the first fixation 48, therefore as long as a second point of vision 38 following a first point of vision 37 remains within the foveal area 34 of the first point of vision 37 and thus within the area of ordered perception of the first point of vision 37, ordered perception is not interrupted and thus continues to fulfil the first fixation criterion 25.
  • first fixation 48 This is therefore a first fixation 48.
  • the first distance 39 is a first viewing angle 41, which preferably describes an area 34 assigned to foveal vision, in particular a radius between 0.5° and 1.5°, preferably approximately 1°, and that the distance between the first point of vision 37 and the second point of vision 38 is a first relative angle 42.
  • FIG. 7 shows a first fixation 48, for example, which is formed from a sequence of four points of vision 37, 38, 69, 70.
  • FIG. 7 also shows the first distance 39, the first viewing angle 41, the first relative distance 44 and the first relative angle 42.
  • each of the four points of vision 37, 38, 69, 70 is a first circle 43 with the radius of the first distance 39, wherein it is clearly shown that the following point of vision 38, 69, 70 lies within the first circle 43 with radius first distance 39 of the preceding point of vision 37, 38, 69, and thus the preferred first fixation criteria 25 is met.
  • the first fixation criterion 25, particularly the first distance 39 and/ or the first viewing angle 41 can be predefined.
  • FIG. 8 shows a viewing sequence in which not all points of vision 37 , 38 , 69 , 70 , 71,
  • the first four points of vision ( 37, 38 , 69 , 70 satisfy the fixation criterion 25 and together form the first fixation 48, wherein the following three points of vision 71, 72, 73 do not satisfy the first fixation criterion 25.
  • Only the fourth point of vision 74 following the first fixation 28 satisfies the first fixation criterion 25 compared to the third point of vision 73 following the first fixation 48.
  • the third point of vision 73 following the first fixation 48 is therefore the first point of vision
  • FIGS. 7 and 8 show illustrative examples, although fixations 48, 49 can occur in natural surroundings with a variety of individual points of vision.
  • the area between the last point of vision 70 of the first fixation 48 and the first point of vision 73 of the second fixation 49 forms a saccade, therefore an area without perception.
  • the angle between the last point of vision 70 of the first fixation 48 and the first point of vision 73 of the second fixation 49 is referred to as the first saccade angle 52.
  • the points of vision 37, 38 assigned to a saccade or a fixation 48 , 49 can now be output for further evaluation, processing or representation.
  • the first and the second point of vision 37, 38 can be output and marked as the first fixation 48 or the first saccade.
  • the following ones are further fixation and saccade definitions that may be used and implemented in the method to mark a fixation event according to the present invention: -Saccades are rapid movements of the eyes with velocities as high as 500° per second, while in fixations eyes remain relatively still during fixations for about 200—300 ms;
  • Fixations are eye movements that stabilize the retina over a stationary object of interest, while Saccades are rapid eye movements used in repositioning the fovea to a new location in the visual environment;
  • the cutoff criterion in this case may be specified in units of angular velocity.
  • a preferred solution is defining the fixation event criterion occurring when eye movements of the gaze tracking device wearer are stabilized over an object of interest (i.e. matches said object of interest) for a predetermined period of time, preferably in the range 0,5 to 5 sec.
  • step 134, 138, 144, 148, 150, 160 it may be implemented that a boundary area around the fixation visual points 21 on the relevant visual representations 10 in order to check if one fixation visual points 21 on the relevant visual representations 10 matches or not the correspondent one predetermined user- object/visible area-related visual point 24, and possibly the following steps mentioned in this specification, as a safeguard alternative.
  • the present invention relates furthermore to a gaze-tracking device 1/a computer device connected to the gaze tracking device 1/the operable device 2 comprising a processor, a computer readable storage medium coupled to the processor said computer readable storage medium having stored thereon computer executable instructions which, when executed, configure the processor to perform the corresponding steps of the method already described in the present specification.
  • Object of the present invention is also the computer readable storage medium having stored thereon computer executable instructions which, when executed, configure the processor to perform the corresponding steps of the method already described in the present specification, according to all the embodiments described and disclosed in this specification.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente invention concerne un procédé basé sur le regard pour déclencher une action sur un dispositif actionnable pourvu d'un dispositif d'affichage. L'utilisateur utilisant un système de suivi du regard peut commander ou déclencher des fonctions spécifiques sur le dispositif actionnable par l'intermédiaire d'informations de regard prédéterminées, mouvement ou regard fixe.
EP21733769.0A 2021-06-14 2021-06-14 Procédé basé sur le regard pour déclencher des actions sur un dispositif actionnable Pending EP4356220A1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2021/065968 WO2022262936A1 (fr) 2021-06-14 2021-06-14 Procédé basé sur le regard pour déclencher des actions sur un dispositif actionnable

Publications (1)

Publication Number Publication Date
EP4356220A1 true EP4356220A1 (fr) 2024-04-24

Family

ID=76532199

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21733769.0A Pending EP4356220A1 (fr) 2021-06-14 2021-06-14 Procédé basé sur le regard pour déclencher des actions sur un dispositif actionnable

Country Status (2)

Country Link
EP (1) EP4356220A1 (fr)
WO (1) WO2022262936A1 (fr)

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8988350B2 (en) * 2011-08-20 2015-03-24 Buckyball Mobile, Inc Method and system of user authentication with bioresponse data
AT513987B1 (de) 2013-08-23 2014-09-15 Ernst Dipl Ing Dr Pfleger Brille und Verfahren zur Bestimmung von Pupillenmittelpunkten beider Augen eines Menschen
US20160231812A1 (en) * 2015-02-06 2016-08-11 The Eye Tribe Aps Mobile gaze input system for pervasive interaction

Also Published As

Publication number Publication date
WO2022262936A1 (fr) 2022-12-22

Similar Documents

Publication Publication Date Title
CN107995979B (zh) 用于对用户进行认证的系统、方法和机器可读介质
US10678897B2 (en) Identification, authentication, and/or guiding of a user using gaze information
CN102567665B (zh) 对无线设备的功能的受控访问
JP2019522278A (ja) 身分認証方法および装置
US20140050370A1 (en) Ocular biometric authentication with system verification
JP6849200B2 (ja) マルチ生体データを用いた非接触式マルチ生体認識方法及びマルチ生体認識装置
CN110114777A (zh) 使用注视信息进行的用户的识别、认证和/或导引
EP4356220A1 (fr) Procédé basé sur le regard pour déclencher des actions sur un dispositif actionnable
KR101993057B1 (ko) 안구 움직임 인식을 이용한 안전한 비밀번호 입력 장치 및 방법
JP2018195141A (ja) 視線の動きによる入室管理システム

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20240114

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

RAP3 Party data changed (applicant data changed or rights of an application transferred)

Owner name: VIEWPOINTSYSTEM GMBH