RU2522848C1 - Method of controlling device using eye gestures in response to stimuli - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: user indicates control positions and creates conditions for the onset of stimuli thereon at a different time. To issue instructions, the user focuses on the control position, waits for the stimulus and, upon seeing the stimulus, makes an eye gesture. User eye movement is detected and analysed and the eye gesture is detected and analysed. At the beginning of the eye gesture, a stimulus in response to which it was issued and the corresponding position are determined, and an instruction associated with said combination of eye gesture and control position is issued to a control device.

EFFECT: enabling detection of user instructions issued by looking without the need for precise determination of spatial coordinates of the look and providing a considerable angular distance between control positions, and without the need for the user to focus at given positions.

21 cl, 2 dwg

Description

FIELD OF THE INVENTION

The invention relates to the field of cognitive technologies, namely to the field of contactless user interaction with controlled devices. It can be used in communication and control systems for computers, robotics, and other technical devices, in particular, intended for people with impaired motor functions, as well as for healthy people in situations where control is preferable to be carried out using eye movements.

State of the art

Traditional methods of transmitting commands from a person to a computer and to a technique controlled by computer technology require the use of finger and hand movements. In this regard, such methods turn out to be inaccessible or inaccessible for a number of diseases, in particular, leading to impaired neuromuscular transmission, in the presence of traumatic injuries of the conduction paths of the spinal cord or amputation of the upper limbs, as well as when the hands are occupied with other activities.

There are other reasons why in certain situations it is preferable to give commands without using limbs. So, giving commands by hand may be undesirable when working with technical devices used by a large number of people, such as ATMs, for hygienic reasons. Providing the ability to issue commands to a computer or smartphone without the use of limbs can increase the entertainment of video games. Entering information into a computer or smartphone without using physical movements of body parts and voice commands can help make it difficult for unauthorized access to confidential information, for example, when entering a password or pin code.

One of the main alternative means by which a person can give instructions to a technical device without using mechanical impact on input devices are technologies based on tracking the direction of the gaze.

When using these technologies, the direction of the user’s view of the positions on the screen that are aligned with the commands (control positions) leads to the issuance of commands (entering letters in the printed text, turning the device on or off, etc.). The disadvantages of these technologies are:

- the difficulty of distinguishing the direction of the gaze to the control position in order to give the team away and involuntarily directing the gaze to the same position (the problem of “touching Midas”);

- the need to provide a significant angular distance between the control positions in connection with the relatively low spatial resolution of this technology (for example, in comparison with the accuracy of cursor positioning with a computer mouse);

- the need for high accuracy of the gaze tracking system, which requires individual adjustment and may require additional adjustment during operation using calibration procedures that create additional load and time for the user.

In these technologies, it is impossible to use the simplest methods of tracking the gaze, which do not require the use of expensive and bulky devices and are based on the registration and analysis of the electrooculogram (EOG) or the video signal recorded by the simplest video cameras, since it is impossible to accurately determine the spatial coordinates of the gaze based on them. Tracking systems using expensive camcorders are commonly used. Ensuring the high accuracy of gaze tracking required when using these technologies involves introducing at least some of the following additional elements into the methodology, which may cause additional inconvenience for the user or limit the possibility of using the technology: using an infrared radiation source to illuminate the pupil and iris region or maintaining a high and stable level of illumination of this area in the visible range; fixing cameras and backlight on the user's head; frequent calibration.

In addition, it is necessary to take into account the possibility of quick fatigue of the user when carrying out the intensively precise oculomotor activity required by these technologies, which is not naturally associated with the provision of the vision function, for which the human oculomotor system is adapted.

Finally, technologies that require accurate positioning of the gaze relative to positions in space (for example, on-screen “buttons”) may not be convenient or accessible for users with impaired vision or impaired gaze control, even when using highly accurate methods for recording eye movements.

To improve the distinction between gaze fixations, related and not related to giving a command, the patents US 4836670, US 4950069, US 4973149 use the criterion of “stop time” (dwell time): the command is given only if the gaze is in a given area of the screen for longer than some predefined time. However, this method does not help to cope with the rest of the above disadvantages of the technology for detecting user commands based on determining the position in which the gaze is directed. In addition, a temporary stop of gaze may not be related to the return of the command: for example, it is possible if the user thinks about something or if something in this area of the screen involuntarily attracted his attention. In this regard, the avoidance of erroneous input of commands when using this method requires considerable effort from the user to follow the instructions and can quickly lead to fatigue. It should also be borne in mind that keeping the gaze in a given position for a given time in order to give a command can be interrupted involuntarily if the user's attention was attracted by something in another position, and long-term fixation of the gaze in one position is difficult and in the absence of pronounced distractions factors. In this regard, the increase in the minimum required stopping time, required to increase the accuracy of distinguishing between attempts to give a command and unintentional fixations, can lead to an even greater increase in the load on the user and even to the inability to control because of the difficulty for a long time to purposefully fix the same spatial position .

In the patent US 6886137 to determine the user's need to slow down or accelerate the speed of movement ("scrolling") of visual information on the screen uses the movement of the gaze, respectively, to the edge of the screen, where the information leaves it, and to the edge where new information appears. Due to the natural nature of such eye movements, they do not create an additional burden on the user. However, this technology is applicable only to a limited range of tasks.

Patent EP 0634031 proposes a method consisting in recording bioelectromagnetic signals generated by the movement of the eyeball, processing them, designating eye movements with symbolic tokens, generating command signals based on a protocol linking command signals with sequences of tokens, and performing communication or control functions in response on command signals. As bioelectromagnetic signals, EOG can be used. The disadvantage of this method is the artificial nature of the use of predetermined sequences of eye movements for issuing commands, due to which fatigue can quickly develop when using the method. In addition, the need to memorize eye movement sequences and their correspondence to commands limits the number of commands used.

Attempts have been made to solve the problems noted above by creating combined eye-brain-computer interfaces. A known method of detecting user-entered commands is based on determining the direction of gaze and analyzing the electrical potentials of the brain associated with fixing the gaze, as well as taking into account changes in pupil diameter and characteristics of eye movements (US 5649061). The disadvantage of this method is the inability to ensure high accuracy of the interface due to the low amplitude of the potentials associated with fixing the gaze, and a significant dependence of the pupil diameter and characteristics of eye movements on various factors that are not controlled or not fully controlled within the framework of this technology.

In a publication by Kotani et al. (2010) (Kentaro Kotani, Yuji Yamaguchi, Takafumi Asao, Ken Horii (2010): Design of eye-typing interface using saccadic latency of eye movement, International Journal of Human-Computer Interaction, 26: 4, 361-376) a method is disclosed , consisting of a combination of detecting a given command by fixing the user’s gaze in the control position corresponding to the command and detecting its subsequent confirmation using an additional saccade. In a specific implementation, the authors used separately located “on-screen confirmation buttons” into which the user had to make confirmation saccades to simplify the identification of such saccades. The method allows to record with great reliability the confirmation event even at low spatial accuracy of the confirming saccades, since the latency of the confirming saccade, which was determined as the time of the beginning of the saccade relative to the time the eye falls into the control position, is used to detect the control position.

Thanks to the use of the confirmation of each input command with an easily detectable saccade, the authors of this publication managed to eliminate errors associated with the “Midas touch” effect. When using their method, there is no need to match each command used with a separate eye gesture (which you need to learn and remember), as in EP 0634031. Since the command designations (in their work are printed letters of the syllable alphabet) can be shown directly in the control positions, the user maybe, in principle, it’s easy to learn how to quickly find the position corresponding to the desired team, even when using different sets of commands.

However, detection of the issued commands in the method proposed by Kotani et al. (2010), requires the precise determination of eye contact in control positions and the provision of a significant angular distance between control positions.

Disclosure of invention

The proposed method for detecting user commands given to a computer using gaze differs significantly from analogues using visual stimuli that provide communication of commands available to a user with known time instants. The user indicates the desired command by submitting an eye gesture: the time of its submission determines the time of appearance of the stimulus corresponding to such a command.

In the simplest case, a single high-amplitude saccade can be used as such an eye gesture. Due to the fact that the latencies of saccades in response to a stimulus (the average value of which is 200 ms in humans) and their variability are relatively small, with the appearance of stimuli at time intervals that are quite large compared to the variability of saccades latency, according to the time of the beginning of saccades in response to a stimulus, determine the stimulus that this saccade responded to. Similarly, it is possible to use more complex eye gestures, especially those that begin with a saccade.

Due to the association of stimuli with commands, the detection of eye gestures and the moments of their beginning allows us to determine the command given by the user. In this case, it is not necessary to obtain accurate information about the position of the gaze relative to the control positions, and this makes it possible to use simple, cheap and reliable methods for recording and analyzing eye movements, including not requiring calibration of measuring equipment.

The proposed method is also aimed at eliminating the need for the user to accurately fix the gaze in the given positions.

In addition, the proposed method is aimed at eliminating the need to withstand significant sizes of control positions, as well as significant angular distances between them. Thus, the present invention creates opportunities for a more free design of control positions and their placement on the screen. Due to this, the interfaces of computer applications using control with the help of a glance, including the interfaces of control systems of equipment connected to a computer can be performed in greater accordance with the tasks for which they are intended, their additional ergonomic optimization is possible, etc.

The technical result achieved by the present invention is to ensure the detection of user commands given to the technical device using the gaze, without the need to accurately determine the spatial coordinates of the gaze and provide a significant angular distance between the control positions, as well as without the need for the user to accurately fix the gaze in the given positions.

The technical result is achieved by the fact that the proposed method of controlling the device using eye gestures in response to incentives, namely, that

- the user is indicated at least one control position,

- at least one eye gesture is associated with each control position,

- with each combination of the control position and the eye gesture associated with it, one team is associated,

- create conditions under which predetermined visual stimuli appear in control positions at different times,

- give instructions to the user to give a command to the device controlled by him, to look into the control position associated with this command, and after the appearance of a given visual stimulus in it, give an eye gesture,

- register the eye movements of the user,

- analyze the recording of the movements of the eyes of the user in order to identify an eye gesture and determine the time of its filing,

- and if an eye gesture is detected, the filing time of which corresponds to the time the stimulus appeared in one of the control positions, the command response associated with the combination of this eye gesture and this control position is detected, and it is issued to the controlled device.

The technical result is also achieved by the fact that

- conditions under which predetermined visual stimuli appear in control positions at different times, are created by presenting specified visual stimuli in these positions at known points in time;

- the presentation of specified visual stimuli in control positions is organized by cycles consisting of the presentation of such a stimulus once in each of at least two control positions;

- if the time moments of the appearance of the given visual stimuli are not known in advance, a signal is recorded that carries information about the appearance of such stimuli, and based on its analysis, the time moments of the appearance of each such stimulus are determined;

- give sound signals synchronized with visual stimuli, while using the same pitch when a given visual stimulus appears in the same position and a different height - when the specified visual stimuli appear in different positions;

- create conditions under which the visual stimuli are the same in all control positions;

- create conditions under which visual stimuli in at least one of the control positions differ from visual stimuli in other control positions;

- the user is shown at least one confirmation position, and the confirming eye gesture is set in the form of a saccade to the confirmation position;

- after the detection of the given command, its designation is shown in the confirmation position;

- the user is shown at least two confirmation positions, and the eye gesture is set in the form of at least two saccades in the confirmation position;

- control positions are placed inside the special area visible to the user, and the user is instructed to direct his eyes inside this special area only when he is preparing to give a command;

registration of the eye movements of the user is performed by video recording of the eye image;

- registration of the movements of the eyes of the user is performed by registering an electrooculogram;

- registration of the movements of the eyes of the user is performed by registering signals of brain origin;

- when analyzing the records of the user's eye movements, the coordinates of the gaze are determined immediately before and after the high-amplitude eye movements;

- register the position of the user's head and the obtained data is used in the analysis of records of the movements of the user's eyes in order to detect an eye gesture;

- along with registration of eye movements, a signal of brain origin is recorded at the user, and the command issued is established using a combined analysis of eye movements and signal of brain origin;

an electroencephalogram is used as a signal of brain origin;

as a signal of brain origin using a magnetoencephalogram;

- a BOLD signal is used as a signal of brain origin;

- A NIRS signal is used as a signal of brain origin.

Brief Description of the Figures

Figure 1 is an example of what a portion of a monitor screen might look like when using a confirmation position. Eight out of nine compactly located control positions are not highlighted 1, one control position is highlighted 2. Confirmation position 3, if used, is located far from the rest of the positions.

Figure 2 shows how control positions can be placed within a visually highlighted special area. In addition to the unselected 1 and 2 selected control positions and confirming positions 3 shown in the previous figure, the border of the special region 4 is shown here, which separates the special region 6 from the rest of the screen 5. Arrows 7-12 show saccades. The direction of saccade 7 excludes its identification as a confirmatory eye gesture. Saccades 8 and 11 in their direction and amplitude can be identified as confirming eye gestures, although they are not. However, the user can exclude them by following the instructions, as to make them, the user must look at a particular area and then make a saccade in the direction of the confirming position - i.e. violate the instruction twice in a row. Saccade 9, which is indeed a confirming eye gesture, is distinguished from saccade 10 with a similar direction and amplitude based on an analysis of their start time (see below). Saccades 12 are possible, to determine the difference between them and confirming eye gestures, it is necessary to use, along with data on the direction and amplitude of the saccade, also data on the coordinates of the gaze in the initial and / or final position, however, it is sufficient to determine such coordinates with relatively little accuracy.

The implementation of the invention

In the most typical case, the method is as follows. The user is located in front of control positions, the position of which is indicated to him visually or verbally. Control positions are associated with predefined commands. To issue a command, the user directs his gaze to the control position associated with this command and expects the stimulus to appear in this position.

In control positions, visual stimuli are presented successively. Having seen the stimulus in the control position associated with the given command, the user immediately gives a predetermined eye gesture.

Record eye movements of the user. By analyzing the registered signal containing information about the movements of the user's eyes, it is determined whether the user gave a predetermined eye gesture upon presentation of the stimulus in one of the control positions, and if the answer is yes, a predetermined command associated with this control position is issued to the controlled device.

Presentation of stimuli, analysis of eye movement records and detection of commands given are carried out automatically using a system that includes a computer program or programs and the necessary technical means. With their help, other operations can be carried out, in particular, the display of control positions.

Further, the above actions are disclosed in more detail and indicating possible options for their implementation.

Management positions

The control position is a place in the space where the incentives associated with one of the teams appear. As control positions, any positions in space can be used, provided that they are all visible to the user.

The control positions make, as a rule, clearly visible to the user in any convenient way, if possible at the same time helping to guess about the teams associated with them or reminiscent of them (see the example in Figure 1). The control position can be designated both as a vast area, and as an area with a minimum extent, at which it can still be visible. Moreover, in the control position itself, there may be no visually distinguishable designation if the surrounding landmarks and instructions issued to the user are sufficient to identify its position. For example, an arrow may indicate a control position. The instruction may indicate that the control position is in the center of the screen or in its upper left corner, while it is permissible not to use any special pointers to indicate it.

 In many cases, it is convenient to designate the control position with an area in the form of a round button with a diameter of 1 ° or a rectangular button of similar dimensions. It should be borne in mind that the purpose of visually noticeable highlighting - control positions in space - to help quickly identify the stimulus associated with the desired team. The most accurate identification of the given command can be expected, in the general case, when the gaze is directed to the control position. However, if necessary, the user can monitor the control position without looking at the control position, provided that it remains in the field of view - this is possible thanks to the mechanisms of hidden attention.

As a device for displaying control positions, you can use the monitor screen of a personal computer, smartphone, etc., as well as the screen on which their designations are projected by a separate projector. However, control positions can be located on any surfaces where the user can see them. As a rule, it is preferable to place all control positions in one plane, however, they can be placed differently, for example, when they are placed on a controlled device of complex volumetric shape, for example, on different "limbs" of an anthropomorphic robot.

Control positions can, in principle, be located at any distance from each other. However, when choosing the distances between them, it should be borne in mind that if they are too close, the stimuli in positions adjacent to the one associated with the input command can interfere with the perception of the stimulus in this position and contribute to fatigue.

If necessary, control positions can change position in space if their designations or landmarks necessary for their location remain visible and their movement does not create significant inconvenience for their location and tracking.

Examples of designations of control positions: pictograms or buttons with short verbal designations on the screen of a monitor of a personal computer, ATM or smartphone; LEDs of different colors mounted in the frame of glasses; color markings applied to fixed or moving parts of a controlled robotic device.

The number of control positions is determined based on the purpose of the system that implements the method. However, it should not exceed 100 and, as a rule, should be significantly lower than this number, in many cases - less than 10. Otherwise, it will be difficult to find the right positions and significantly slow down management, because the time required to return one team increases with the number of positions (approximately proportional to it).

Incentives

As an incentive, any event visually noticeable to the user can be used, the time of which is known or is available for registration in a software system that implements the invention, and the association of which with one of the control positions is known to the user.

The user should be made aware of the type of incentive used, i.e. he should be informed of what kind of events he must react to give a command if it does not become "intuitive" immediately immediately when observing the stimuli.

In the most typical case, a change in all or most of the designation of the control position is used as a stimulus (cf. 1 and 2 in FIG. 1). As an incentive, you can also use a change that is well noticeable to the user: an area less extensive than the designation of the control position, and located within this designation; an area partially overlapping with the designation of the control position; area more extensive than the designation of a controlling position, and including it in itself. It is acceptable to use as a stimulus a change in an area that does not overlap with the designation of the control position if the association of the stimulus and the control position remains clear to the user - for example, when the distances between the stimulus and the control position are much smaller than the distances between different control positions.

In the most typical case, a short-term change in brightness and / or color is used as a stimulus. In this case, two events actually occur, the first of which consists in a change in visual characteristics in comparison with the initial ones, i.e. with those that are observed most of the time, and the second is in the return of visual characteristics to the original. It is advisable to use only the first of them as an incentive, however, it is customary to talk about the "duration of the incentive", i.e. about the time passing between such paired events.

The duration of the stimulus is chosen so that the user can notice a change, but the duration of the stimulus should not be too long, otherwise the next stimulus must be presented before the end of the previous one, or the procedure will become excessively long.

In other cases, the event used as a stimulus may not be accompanied by a return to the original visual characteristics: for example, it may be a color change that lasts all the time until the next stimulus, or rotation of an elongated rectangle 90 ° around its center. As an incentive, you can use the moment of the start of movement or the moment of stopping of the robotic arm or virtual object on the computer screen.

The changes used as incentives, as a rule, are of a uniform nature for the same control position when they appear repeatedly, as well as for different control positions, including if the control positions differ significantly in their visual characteristics. For example, as incentives, you can use an increase in brightness by the same amount, an increase in the size and scale of the icon by the same amount, etc.

In certain cases, especially when the control positions are close to each other and areas where the visual change is used as stimuli appearing in adjacent control positions are overlapped, to simplify the identification of the stimulus appearing in the position associated with the desired command, it may be appropriate to use a different change in different positions.

It is preferable to inform the user in advance what the incentives look like, or to give them the opportunity to familiarize themselves with how they look before using the method.

To facilitate the determination of the moment of the stimulus associated with the desired control command, one can accompany the sound signal with the same height by the supply of stimuli associated with the same control position and sound signals with different heights - stimuli associated with different control positions. This modification may be useful, in particular, for people with impaired vision.

As a rule, specially organized events, the time of which is known in advance in the software system, are used as stimuli. However, an event whose time could not be known in advance can also be used as an incentive, provided that its time is recorded in the software system and that the user can identify such an event as a stimulus. Examples of such events include the contact of a moving robotic arm with an object and the combination of the sight with a predetermined goal in the game. Since a large number of different events can occur in the space in front of the user, to use any of them as stimuli, the signs that the event must correspond to in order to be considered a stimulus are determined in advance. These signs inform the user, and also provide registration and analysis of the signal for their automatic identification.

Regardless of whether the presentation of stimuli is organized by the program or only programmatically records the time of occurrence of stimuli, any two stimuli, as a rule, should not overlap in time or converge in time to such an extent that it is difficult to perceive any of these stimuli.

The duration of stimuli and pauses between them can be both constant and varying in time. It is preferable to install them in the process of testing a specific implementation of the proposed method and taking into account its features, as well as the features of the intended users, achieving easy and error-free recognition of stimuli and low variability of latency of eye gestures in response to stimuli. It is also possible to use information on a person’s perception of stimuli of the corresponding type and on the variability of saccade latency in response to stimuli known from the literature on experimental psychology. To accurately determine the command to be given, the total duration of the stimulus and pause should be set higher than the spread (the difference between the maximum and minimum values) of latency of eye gestures in response to stimuli.

In particular, for healthy users, in most cases it may be convenient to use a stimulus duration of 200 ms and a constant or minimum pause between stimuli of 100 ms duration. If the system is intended to be used by severe patients, the duration of the stimuli and the pauses between them should be extended.

The order of presentation of incentives in control positions can be either random or predetermined, including always the same. For example, if the available commands are entering numbers from 1 to 9, it is preferable to present incentives in the order of control positions with the designations 1, 2, ... 9, then again 1, 2, etc. When implementing the analysis of the user's eye movements (discussed below), it is taken into account that in this case the moment of presentation of the stimulus in the position associated with the given command is predictable for the user, and in this regard, the confirming eye gesture can have very low and even negative latency.

In some cases, for example, if it is necessary to complicate unauthorized access by unauthorized persons to information entered by the user (for example, a PIN code), as well as when using, in addition to recording the user's eye movements, also signals of brain origin, amplifying with increasing uncertainty of presentation time stimulus, it may be preferable to use a random presentation of stimuli.

It is possible to exclude the presentation of incentives in some control positions (excluded control positions), for example, if the commands associated with them are currently impossible to execute.

The presentation of stimuli can be organized by stimulus cycles - sequences consisting of the presentation of stimuli once in each of the used (not excluded) control positions. Additional pauses may be made between cycles.

It may turn out to be convenient - especially when using long pauses (for example, more than 3 s) - to give an audible or visual signal before the start of each cycle, warning the user that stimuli will begin soon. Such a signal - a warning signal - can be given, for example, 500 ms or 1 s before the start of the cycle.

The presentation of stimuli, with pauses between them and, if necessary, with pauses between cycles, can be performed both during the entire time the proposed method is used and within limited time intervals. Presentation of a given number of stimulus cycles may also be arranged.

The ability to trigger presentation of incentives may be provided to the user. For example, after 500 ms after each press of a button, one stimulus cycle can be launched on the device, while the use of manual input is reduced to using a single button to indicate moments of readiness to enter a command, and the actual choice of several commands is made using eye gestures. In other embodiments, the user can use a separately defined eye gesture, a mechanical gesture (for example, a specific movement of a finger along the touch surface or in the touch space) to trigger stimuli, or a voice command.

The presentation of incentives can be interrupted immediately after the detection of the return of the command. If it is necessary to impede access to the input information, it is preferable to continue to present stimuli even after detection of the return of the command, at least until the end of the cycle. If stimuli, as a rule, are given in cycles without pauses between them, after detecting a given command, you can pause between the end of the current cycle and the beginning of the next cycle. If stimuli, as a rule, are given in cycles with small pauses between them, after detecting a given command, you can extend the pause between the end of the current cycle and the beginning of the next cycle.

Eye gestures and ways to improve their recognition for eye movement analysis algorithms

An eye gesture is understood as an eye movement or a sequence of eye movements, with which the user marks the time when he found the stimulus applied in the control position associated with the desired command to the user. By the time of the beginning of the eye gesture, it is possible to determine which stimulus it corresponds to. Thus, by giving an eye gesture, the user reports which command he wants to give.

The user is preliminarily instructed to give a command to direct a look at the position associated with the team that he wants to give, wait for the stimulus in this position, and immediately make an eye gesture after it appears. About what an eye gesture should be, he is also informed in advance.

More than one eye gesture is possible. In this case, a separate command can be associated with each combination of an eye gesture and a control position, similar to how it is done with combinations of ordinary gestures and touch areas to provide a significant angular distance between the control positions in RU 2455676.

Eye gestures are selected taking into account the requirements for a specific implementation of the invention: firstly, the requirements of minimizing the error in choosing a team, and secondly, the requirements of minimizing the frequency of false alarms.

To minimize the frequency of false alarms, it is necessary that an eye gesture, intentionally given by the user in order to give a command, when analyzing records of eye movements can be distinguished from other eye movements and eye movements, especially spontaneous ones. To fulfill this requirement, eye gestures are preferred, as rarely as possible with spontaneous oculomotor activity, for example, a sequence of large saccades in different directions.

At the same time, to minimize the frequency of team choice errors, it is necessary that, as a result of analyzing the recordings of eye movements, the eye gesture is correctly associated with the particular stimulus in response to which it was given, and not with another stimulus. For this, the latency spread of the onset of the eye gesture should be as small as possible. In this regard, as well as to reduce the load on the user, the most simple eye gestures are preferred. The use of simple eye gestures also allows you to increase the speed of response, since they have a shorter duration.

An important special case of an eye gesture is a saccade to a predefined position known to the user - a confirmation position (a technique used by Kotani et al. (2010) to recognize confirmatory saccades from control positions, but without using incentives). When using confirming eye gestures in the form of such saccades, the instruction for the user may include, for example, such an item (when using highlights as stimuli):

“Wait until the control position is highlighted and, having seen its illumination, immediately turn your eyes to the confirming position.”

The value of eye gestures in the form of a saccade in a confirming position is due to the fact that they can combine the simplicity and significance of differences from eye movements that are not related to control. Thus, their use can help reduce both the frequency of false alarms and the error rate of team selection.

Like control positions, the confirming position can be located on various surfaces where the user can see it, and can be indicated in various ways, ensuring their visibility to the user and, if possible, helping to remember their purpose.

The confirming position is preferably located separately from the control positions. When choosing a position for her, it is based on the fact that the farther she is from the control positions, the easier it is to distinguish the saccade in it from eye movements that are not associated with confirmation of the command, especially in the absence of the ability to determine the position of the gaze with high accuracy. At the same time, they also take into account the compactness requirements for placing user-visible elements in a particular implementation.

At the same time, you can use more than one confirming position - for example, if looking in one of the confirming positions in some situations is undesirable because of the possibility of losing visual control over any visible area of the screen or the space around it, as well as when using more than one eye gesture.

Figure 1 is an example of what a portion of a monitor screen might look like when using a confirmation position. Confirmation position 3 is usually located at such a distance from other positions that the user sees the distance between the nearest edges of the confirmation position and the closest control position to it at an angle from 2 ° to 10 °. In this case, saccades used as confirming eye gestures have a significant amplitude, and due to this, their detection is improved and the number of saccades unrelated to command return, which cannot be distinguished from control eye gestures by their characteristics, is reduced.

If the absolute coordinates of the gaze are tracked with low accuracy, to improve the discrimination of saccades in the control position, you can place the control position inside the user-specific area (see Figure 2). The distance from the control positions to the border of a particular area is selected so that it is possible to distinguish saccades from outside this area from saccades from control positions to a confirming position, knowing the amplitude and direction of saccades with the accuracy provided by the used technology for recording and analyzing eye movements. The user is advised to avoid looking at a particular area followed by a saccade toward a confirmation position, except when using saccades from a control position to issue a command.

Another important way to significantly reduce the frequency of false alarms can be to turn on the system only for a short time.

In particular, for users with preserved limb motility, it may be convenient to put the system into an active state only for a short time using normal motor actions, for example, by pressing a button specially provided for this or by issuing a voice command. Prior to this action, the system is in a passive state, i.e. no incentives are presented and the system software does not try to establish the fact of issuing commands, except for the command to bring the system into an active state. Due to this, in the passive state of the system, the load on vision from the side of stimuli is eliminated and false alarms are simultaneously eliminated. The presentation of stimuli begins after the user gives the command to bring the system into an active state, having stood a short pause (for example, 500 ms) before initiating the stimulus delivery, and, as a rule, they present one cycle of stimuli. The user has the opportunity in advance - before bringing the system to an active state - to find the desired control position and to avoid large saccades that are not related to control during one cycle of presenting incentives. Due to this, false alarms can be almost completely eliminated.

After the presentation of one cycle or a given number of cycles of stimuli, the system is transferred to a passive state.

An early return of the system to a passive state may be provided with an additional command. In this case, it is advisable to cancel the command if its return was detected for a certain time before the system was put into a passive state, for example, 200 ms before. This can be useful in cases where the user decides not to give the command, or if he finds that he has made a saccade, similar to the confirmation one, not after presenting the stimulus in the position corresponding to the team he needs, or with a delay after such presentation.

A control method using conventional motor actions and stimuli could be implemented without the use of eye movements (for example, the user stops pressing the button immediately after he sees the stimulus in the desired control position), but this eliminates the important technical result achieved in the proposed invention, namely the ability to significantly complicate third-party observers access to information about which commands the user enters.

The system can be switched to an active state for a short time with a predefined eye gesture.

Finally, the system can be automatically switched to a short state automatically, for example, at a time known to the user (at the beginning of each minute, while the user sees the second hand of the clock or the current time in seconds in a numerical representation) or when an event occurs after which team selection may be required (wheelchair stops after automatically detecting an obstacle - the user is given the opportunity to choose an obstacle detour on the left or right). In these cases, it may be useful to warn the user about the upcoming turn on of the system by means of an alarm signal given a short time before it is turned on, for example, 1 s before it.

It is possible to use various combinations of methods for transferring the system to an active state. In particular, when using the proposed invention in ATMs in order to exclude physical contact of the user with the external surfaces of the ATM and / or make it difficult for unauthorized persons to access confidential information, the first activation of the system occurs some time after the user inserts a bank card in the slot for it, and then activation occurs after giving users the specified eye gesture or after the specified time after the previous command (the user is preceded each time to warn of a particular method of the system into an active state).

Instead of a confirmation position, you can use a predefined direction of confirmation. For example, the user is given instructions after he sees the stimulus in the position associated with the desired team, immediately look up. To highlight the direction of confirmation, especially if it is not easily remembered - for example, in the 30 ° direction clockwise relative to the upward direction - it is advisable to use markers that are noticeable to the user. The user is informed that he should not hold his gaze on such a marker, but instead should move his gaze much further away to the extreme position of deviation of the gaze in the same direction or to a position close to such an extreme position. It is possible to use both the direction of confirmation and the confirming position: when the stimulus appears in the control position, the user is first offered to make a saccade in the direction of the confirmation position, but as far as possible, and then return the look to the confirmation position.

In some cases, it is not saccades that can be used as an eye gesture, but tracking eye movements fixed on a predetermined moving object, or on an object moving in a given direction, or on the object with which the execution of a given command is associated. For example, stimuli can be presented on a transparent screen through which moving parts of the robot are visible. Having seen the stimulus in the position associated with the desired team, the user immediately looks at the moving part of the robot to which the given command is to be applied, and fixes his gaze on it, tracing its movement, for a short time interval (for example, within one second) .

As an eye gesture, the sequence of any of the eye gestures described above, including the same ones (but, as a rule, without returning the gaze to its original position). For example, more than one confirmation position is created on the monitor screen, and the user is instructed as a confirmation eye gesture to make a saccade to the first confirmation position and then to other confirmation positions in the specified order. When using tracing eye movements as an eye gesture, one can consider a saccade on a moving object preceding fixing the gaze on it, as part of an affirming eye gesture.

It is preferable to choose sequences of eye gestures that can be done spontaneously as little as possible, without trying to give a command. Moreover, the frequency of false alarms can also be significantly reduced.

Automation of the return of eye gestures can contribute to increasing accuracy and speed of work, which, in turn, can facilitate the natural nature of eye gestures. In some cases, it may be useful to use different eye gestures for different teams. For example, giving a command to a wheelchair to turn left can be done using a saccade to the left, turn right - using a saccade to the right.

When saccades are used as eye gestures in a single confirmation position, the preferred method of facilitating, stabilizing and, if possible, automating eye gestures is to display directly in or near this position information about the command being given (for example, its verbal or graphic designation). This method should be effective due to the fact that the translation of the look into a position where important information can be obtained at the moment is the most natural.

Automation of saccades to the confirmation position can also be facilitated by showing the designation of the command in the confirmation position along with the start of highlighting the corresponding control position, with the continuation of its display in case of detection of the return of this command. In this case, the confirmation position is also the place where feedback is received, but, in addition, the presence of the designation of the input command in it, especially if there is such a designation and in the corresponding control position, can help to combine this control position and the confirmation position into a single perceptual or semantic "Gestalt", and this, in turn, can contribute to a more stable and faster confirming saccades.

The implementation of the method may include the possibility for the user to independently appoint an eye gesture before issuing commands. To do this, at least one control position is created and the user makes his preferred eye gesture in response to a stimulus that appears in a single control position or in one of the control positions. In this case, the movements of the user's eyes are recorded and analyzed. It is preferable to register eye gestures in response to at least several stimuli. An analysis of the recorded eye movements as part of an eye gesture may include, in particular, highlighting a recording segment corresponding to an eye gesture, highlighting the eye movements from which it is composed, and training the classifier to recognize an eye gesture based on the combination of characteristics of eye movements that make up the eye gesture.

Recording eye movements, analyzing eye movement records, recognizing eye gestures and giving commands

To date, the prior art knows a large number of methods for recording and analyzing eye movements, recognition and analysis of saccades, tracking eye movements, as well as eye gestures. When implementing the invention, any of these methods that best suit the objectives of a particular implementation can be used.

As a rule, video recording of eye images using a video camera is used to record eye movements, and it is permissible to use cameras with a relatively low resolution, such as webcams. In some cases, it may be more convenient to register an electrooculogram.

If the registration of eye movements alone is insufficient to distinguish between confirming eye gestures and spontaneous gestures with the reliability required in a particular implementation of the method, or if it is desirable to increase it, tracking of the registration of eye movements by registering the position of the head is possible. In this case, recordings of eye movements and head position are analyzed together.

If the used method of recording eye movements allows you to at least approximately determine the coordinates of the gaze, it is useful to determine these coordinates immediately before and after high-amplitude eye movements and use this information to detect confirming eye gesture. With a sufficiently high accuracy of determining the coordinates of the gaze, this information can also be used to increase the accuracy of detection of the given command.

When using methods for recording eye movements that provide only low accuracy in estimating the coordinates of the gaze or only changing these coordinates (which is typical, for example, for the use of EOG), special attention is paid to the selection of confirming eye gestures and the implementation of the method as a whole in this way (for example, using the inclusion of the system only for a short time) so that confirming eye gestures can be distinguished by recordings of eye movements from the vast majority of spontaneous eye movements.

As leading signs of confirming eye gestures, as a rule, it is advisable to use the correspondence of the directions of their constituent eye movements, as well as their amplitude, to the expected directions and amplitude. If more than one confirming eye gesture is specified, after detecting and analyzing the eye gesture, it is determined which of the specified eye gestures was used. If more than one control position is used, the determination of the stimulus in response to which an eye gesture was given and the control position associated with it is made.

The expected direction and amplitude and, if possible, the coordinates of the beginning and end of eye movements that make up eye gestures are determined based on the expected orientation of the user's head relative to the control positions and when they are used, confirmation positions and from the distance from the user's eyes, as well as based on the description of the specified eye gestures. If possible, these data are corrected in the process of using the method.

To determine the stimulus in response to which an eye gesture was given, the start time of the gesture is corrected (reduced by subtraction) by approximately 200 ms and, if possible, calculated more accurately or in a study using this implementation of the method on a group of subjects, which is sufficient a representative sample of the estimated population of potential users, either in a trial use of the method by a specific user, or otherwise. The obtained corrected value indicates the estimated time of the beginning of the stimulus, in response to which this eye gesture was given. A comparison is made of this time with the recorded time of stimuli that appeared, for example, during the last cycle of stimuli or during some fixed time before the beginning of a given eye gesture (for example, no earlier than 500 ms before its start) and find the stimulus whose start time is the most close, in comparison with other stimuli, to a given adjusted start time of the confirming eye gesture.

If it is possible to accurately determine the coordinates of the gaze, they are additionally used both for identifying an eye gesture and for identifying the control position to which the gaze was directed before applying the eye gesture.

Using signals of brain origin

Hybrid technologies are currently being developed that combine the brain-computer interface and methods using gaze-based control (e.g., Antonio Frisoli et al. (2012): A new gaze-BCI-driven control of an upper limb exoskeleton for rehabilitation in real -world tasks, IEEE Transactions on Systems, Man, And Cybernetics, Part C, Vol. 42, No. 6, pp. 1169-1179). The proposed method allows for natural combination with a known variety of non-invasive brain-computer interfaces, which use stimulus reactions (US7546158, US8155736). With this combination, the same stimulus provokes the user's response in the form of a confirmation eye gesture and a characteristic pattern in signals non-invasively recorded from his brain (signals of brain origin), indicating increased attention to this stimulus in comparison with other stimuli. Due to this, the accuracy of determining the stimulus corresponding to the desired user command can be improved, and hence the accuracy of detection of the given command.

The most practical signal of brain origin for use in the proposed method is currently an electroencephalogram (EEG). However, significant progress has been made in technologies for recording signals of brain origin (see, e.g., T.N. Sander et al. (2012), Magnetoencephalography with a chip-scale atomic magnetometer, Biomed. Opt. Express. Vol. 3, No. 5, pp. 981-990; S. Xu et al. (2006) Magnetic resonance imaging with an optical atomic magnetometer, Proc. Natl. Acad. Sci. USA. Vol. 103, No. 34, pp. 12686-12671; Dmitry Budker, Michael Romalis (2007) Optical magnetometry. Nature Physics, Vol.3, pp.227-234), and in the future it may be appropriate to use, along with or instead of EEG, also magnetoencephalograms (MEG) and / or BOLD signal, obtained by magnetic resonance. A potentially promising signal of brain origin for use in the proposed method is also a signal obtained using near infrared spectroscopy (NIRS).

A signal of brain origin can be used both in addition to data on eye gestures in response to stimuli, and to obtain data on eye fixations or to refine them based on the method proposed in patent US5649061.

Example 1

The proposed method can be used to enter a personal secret code (pin code) when entering the room in such a way as to exclude access to the pin code of unauthorized persons, and also to exclude the touch of hands on any surfaces. Control positions are indicated by placing the numbers from 0 to 9 on an 8x8 cm screen. Visual stimuli are presented by LED illumination of these numbers in random order for 200 ms with pauses of 100 ms between the backlights, the cycles consisting of the backlighting of each digit once. To start the presentation of stimuli, the user brings an electronic pass (contactless radio frequency card) to the reader, in case of positive identification, a short beep is given and after 500 ms the first cycle of presenting stimuli begins. Between the end of each cycle (except the last) and the beginning of each subsequent cycle after a pause of 500 ms, an audio signal is also given and a new cycle begins 500 ms after it. Using eye gestures, in response to stimuli, the user must sequentially enter four digits that make up the PIN code - one per stimulus cycle. To do this, having seen the backlight of the number in the pin code at the position corresponding (in order) to the current cycle number, the user makes an eye gesture in the form of a quick look in the direction indicated in advance, for example, to the left, and returns to the screen.

Eye movements are recorded using a video camera installed near the screen. It is possible to use eye illumination using an infrared source. Video footage of eye movements is analyzed at time intervals from the first stimulus to the last stimulus included in each cycle, plus 300 ms after the end of the last stimulus. When analyzing video recordings, the position of the eyes is determined, the iris and pupils are identified, and their borders are outlined. A possible supply of a given eye gesture is detected based on the analysis of changes in the direction of view. Within each cycle, determine the eye gesture, the most similar to what is expected from the user, and the time of the start of his filing. 200 ms are subtracted from this time and a stimulus is found, the time of which was closest to the obtained value. The user identification device is instructed to enter a digit associated with this stimulus in the sequence of input digits. After the length of this sequence becomes equal to four digits, it is compared with the pin code of the given user, in case of coincidence, the user identification device issues permission to pass (open the corresponding lock, etc.), and in case of discrepancy - an error message.

Example 2

When using the proposed method in a wheelchair control system in a situation when it approaches an obstacle, conditions for the appearance of incentives can be created by projecting a light spot (for example, a laser pointer) two meters from the wheelchair in front of him along the path of his movement. The control position in this case is the position of the light spot. A given visual stimulus is the hit of such a light spot on an obstacle, a collision with which is undesirable and which protrudes above the surface on which the movement is carried out.

The user is informed that upon the occurrence of such an event, one of three commands can be given to the wheelchair: drive around the obstacle on the left - by giving an eye gesture in the form of a short-term translation of the gaze to the extremely left position; go around the obstacle to the right - by filing an eye gesture in the form of a short-term translation of the gaze to the extreme right position; to stop - by giving an eye gesture in the form of a short-term translation of the look down (In the absence of detection of a command, the chair is automatically controlled in accordance with the results of the analysis of the nature of the obstacle and based on a given program.)

The registration of the user's eye movements is carried out by registering a video image of his eyes with a video camera mounted on a wheelchair. It is possible to use eye illumination using an infrared source. When analyzing video recordings, the position of the eyes is determined, the iris and pupils are identified, and their borders are outlined. Using existing means of analyzing the coordinates of the gaze, it is approximately determined whether the gaze is directed toward the light spot. A possible supply of a given eye gesture is detected based on the analysis of changes in the direction of view.

Using a distance sensor to objects on the path of a wheelchair (for example, integrated with a laser source of a light spot), a signal is received that carries information about the appearance of the stimulus, and by analyzing it the moments of time of the appearance of the stimulus are determined - i.e. detect times when a light spot hits a potential obstacle protruding above the surface. To improve the identification by the user of the moment the stimulus appears, immediately after the detection of a light spot hitting an obstacle, the system automatically issues a sound signal.

After that, if the gaze was directed in the same direction where the light spot was, and one of the specified eye gestures was detected with a start in the interval -100 ... + 300 ms relative to the beginning of the sound signal (take into account the possibility of advanced detection of a light spot on an obstacle ), determine whether it was an eye gesture in the form of a glance to the left, right or down, and give the wheelchair a command associated with this eye gesture. An eye gesture is also detected if the gaze before the sound signal was not directed towards the light spot, but was shifted approximately in its direction 150 ... 300 ms after it was given, and immediately after that the specified eye gesture was recorded.

Claims (21)

1. A method of controlling a device using eye gestures in response to stimuli, comprising the step of indicating to the user at least one control position, at least one eye gesture associated with each control position, with each combination of the control position and an ocular associated with it one command is associated with the gesture, create conditions under which the given visual stimuli appear in the control positions at different times, instruct the user to transfer the command to the device controlled by him to looking into the control position associated with this command, and after the specified visual stimulus appears in it, give an eye gesture, register the user's eye movements, analyze the records of the user's eye movements in order to detect an eye gesture and determine the time of its filing, and if an eye is detected a gesture, the feed time of which corresponds to the time the stimulus appeared in one of the control positions, detect the return of the command associated with the combination of this eye gesture and this control position, and in give it to the controlled device. 2. The method according to claim 1, characterized in that the conditions under which predetermined visual stimuli appear in control positions at different times are created by presenting predetermined visual stimuli in these positions at known points in time. 3. The method according to claim 2, characterized in that the presentation of specified visual stimuli in control positions is organized by cycles consisting of the presentation of such a stimulus once in each of at least two control positions. 4. The method according to claim 1, characterized in that if the time instants of the appearance of predetermined visual stimuli are not known in advance, a signal is carried that carries information about the occurrence of such stimuli, and the time instants of each such stimulus are determined based on its analysis. 5. The method according to claim 1, characterized in that they emit sound signals synchronized with visual stimuli, using the same pitch when a given visual stimulus appears in the same position and a different height when specified visual stimuli appear in different positions. 6. The method according to claim 1, characterized in that they create the conditions under which the visual stimuli are the same in all control positions. 7. The method according to claim 1, characterized in that they create the conditions under which visual stimuli in at least one of the control positions differ from visual stimuli in other control positions. 8. The method according to claim 1, characterized in that the user is shown at least one confirmation position, and the confirming eye gesture is set in the form of a saccade to the confirmation position. 9. The method according to claim 8, characterized in that after the detection of the given command, its designation is shown in the confirmation position. 10. The method according to claim 8, characterized in that the user is shown at least two confirmation positions, and the eye gesture is set in the form of at least two saccades in the confirmation position. 11. The method according to claim 8, characterized in that the control position is placed inside the special area visible to the user, and the user is instructed to direct his eyes inside this special area only when he is preparing to give a command. 12. The method according to claim 1, characterized in that the registration of the movements of the eyes of the user is performed by video recording of the image of the eyes. 13. The method according to claim 1, characterized in that the registration of the movements of the eyes of the user is performed by registering an electrooculogram. 14. The method according to claim 1, characterized in that the registration of eye movements of the user is performed by registering signals of brain origin. 15. The method according to claim 1, characterized in that when analyzing the records of the user's eye movements, the coordinates of the gaze are determined immediately before and after the high-amplitude eye movements. 16. The method according to claim 1, characterized in that the position of the user's head is recorded and the obtained data is used in the analysis of records of the user's eye movements in order to detect an eye gesture. 17. The method according to claim 1, characterized in that along with the registration of eye movements, the user records a brain signal, and the command issued is established using a combined analysis of eye movements and a brain signal. 18. The method according to 17, characterized in that an electroencephalogram is used as a signal of brain origin. 19. The method according to 17, characterized in that as a signal of brain origin using a magnetoencephalogram. 20. The method according to 17, characterized in that as a signal of brain origin using a BOLD signal. 21. The method according to 17, characterized in that the NIRS signal is used as a signal of brain origin.

Images