WO2019144196A1

WO2019144196A1 - Eye-typing method, system and computer readable medium

Info

Publication number: WO2019144196A1
Application number: PCT/AU2019/050058
Authority: WO
Inventors: Jordan Son Nguyen; Nicholas John Temple
Original assignee: Psykinetic Pty Ltd
Priority date: 2018-01-25
Filing date: 2019-01-25
Publication date: 2019-08-01
Also published as: AU2019210707A1

Abstract

Disclosed is a computer implemented method for enabling eye-typing. The method includes receiving a stream of gaze data including gaze samples directed toward a virtual keyboard of a user interface presented by a display device, and determining that the user has eye-typed a word by detecting, based on the stream of gaze data, that the user's gaze has moved to a text presentation portion of the user interface. In response to detecting that the user's gaze is located within the text presentation portion of the user interface, the method includes analysing the stream of gaze data received prior to the detection to determine the word which the user has eye-typed. The method further includes presenting the determined word in the text presentation portion of the user interface, and determining that the user has begun eye-typing a further word based on a later portion of the stream of gaze data indicating that the user's gaze has left the text presentation portion of the user interface. Also disclosed is a computer implemented method for launching a calibration session for an eye-typing system.

Description

EYE-TYPING METHOD, SYSTEM AND COMPUTER READABLE MEDIUM

Technical Field

[0001] The present invention relates to a computerised method, computerised system and computer readable medium for high speed eye-typing.

Background

[0002] Eye-typing refers to the activity of typing text using an eye-tracker, often for usage within a communication application. Eye-typing can be used by anyone who is able to use an eye-tracking system, but most importantly provides an alternative means of communication for people with profound physical disabilities who require communication assistance. Eye typing is the primary method of communication for many individuals with less severe motor impairments as well, as eye control is more rapid and less fatiguing than other functions for people with this form of disability.

[0003] The interaction methods for eye-typing applications have been implemented in a multitude of ways with many different interfaces in the literature.

[0004] One of the most common techniques is dwell-based systems which involved detecting that the gaze of a user had“dwelled” for a period of time over the location of a particular key of a keyboard presented by a display device. Dwell times can range between 200-600ms and generally up to a maximum lOOOms. Less experienced users generally require a longer dwell time which results in a low rate of eye-typing (e.g. 12 words per minute (WPM)). For more experienced users, the dwell time can be reduced in order to increase the rate of eye-typing to a rate of 20-24 WPM. However, this also results in a problem known as the‘Midas Touch’ which refers to the challenge of building a user interface that minimizes the effect of accidental gaze interactions. More specifically, it is not possible to determine user intention behind raw gaze data from positioning alone with 100% accuracy. [0005] Dwell-free eye-typing systems have been proposed. In particular, such a system relies on gestures performed by a user over a key, consisting of looking inside the key, outside the key and then inside the key again. However, the problem with such a system is that it relies on an unnatural gesture. Although dwell-time is not required, three fixations are required for any given key selection resulting in a significant amount of time performing this gesture, resulting in eye-typing speeds similar to those achieved with dwell-based systems.

[0006] Another dwell-free eye-typing technique involved detecting gaze samples over keys of a keyboard, wherein characters were filtered out by use of a corpus. Similar eye-typing speeds were achieved to those discussed above. Such a method is clearly heavily dependent on the corpus and involves a large amount of uncertainty where the user quite heavily relies on the word- suggestions for typing and the delay involved.

[0007] As such, eye controlled communication software takes significantly more time than it should for the user to input the words they wish to communicate or type.

[0008] Another problem concerns initiation of calibration routines of current eye-typing systems. Calibration is necessary to improve eye-typing accuracy by adjusting a response of an eye-typing system to gaze characteristics of a specific user. The need for calibration normally arises when there is some inaccuracy or inconsistency in the eye-typing system’s ability to track a user’s gaze. Therefore, eye-typing systems should provide ways to initiate a calibration routine other than in response to a user’s gaze commands.

[0009] Some eye-typing systems provide the option of initiating a calibration routine through a manual command, such as by pressing a button on a controller or by using a mouse. However, this arrangement is inconvenient for some users with severe motor impairments, who must wait for assistance from an able-bodied person to commence execution of the calibration routine. [0010] A solution to this problem includes eye-typing systems that are configured to automatically launch a calibration routine, for example, at pre-determined times during operation of the eye-typing system. This, however, can lead to the eye-typing system launching a calibration routine even at times when calibration is not necessary, which is a nuisance to the user because calibration routines can last for several minutes, and therefore interrupt the user, impairing their eye-typing experience.

[0011] There is a need for new or improved systems for eye-typing and/or methods or processes for enabling eye typing.

[0012] The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as, an acknowledgement or admission or any form of suggestion that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

Summary

[0013] In one aspect there is provided a computer implemented method for enabling eye typing including steps of:

receiving a stream of gaze data including gaze samples directed toward a virtual keyboard of a user interface presented by a display device;

determining that the user has eye-typed a word by detecting, based on the stream of gaze data, that the user’s gaze has moved to a text presentation portion of the user interface; in response to detecting that the user’s gaze is located within the text presentation portion of the user interface, analysing the stream of gaze data received prior to the detection to determine the word which the user has eye-typed;

presenting the determined word in the text presentation portion of the user interface; and determining that the user has begun eye-typing a further word based on a later portion of the stream of gaze data indicating that the user’s gaze has left the text presentation portion of the user interface.

[0014] In certain embodiments, the step of analysing the stream of gaze data includes:

determining, based on the gaze samples obtained at a predefined sampling frequency, a sequence of gaze velocities;

determining, based on the sequence of gaze velocities, one or more keys of the virtual keyboard corresponding to a respective one or more gaze velocity troughs; and

determining, using the one or more keys, the word.

[0015] In certain embodiments, prior to the determining the sequence of gaze velocities, the method includes smoothing, using a smoothing function, the plurality of gaze samples.

[0016] In certain embodiments, the smoothing function is a kernel smoother.

[0017] In certain embodiments, the method includes:

determining, based on the plurality of gaze samples, a sequence of vectors corresponding to a movement path of the user’s gaze;

determining, based on the sequence of vectors, a sequence of angular displacements between successive vectors from the sequence of vectors; and

determining the word eye-typed by the user by searching a data store of words based on a search criteria indicative of the sequence of angular displacements, wherein each word in the data store has a respective sequence of angular displacements.

[0018] In certain embodiments, prior to the search of the data store, the method includes filtering the sequence of angular displacements using an angular displacement threshold, wherein each angular displacement less than or equal to the angular displacement threshold is discarded and one or more subsequent angular displacements are adjusted according to the discarded angular displacement. [0019] In certain embodiments, the method includes determining a start of an eye-typing session based on detecting a user’s gaze being fixated upon a selected key from the virtual keyboard for a period of time that is greater than or equal to a dwell threshold, wherein the selected key represents a first keystroke that has been eye-typed by the user for the word.

[0020] In certain embodiments, prior to the determining the start of the eye-typing session and in response to determining that the user’s gaze is fixated upon the selected key from the virtual keyboard, the method includes dynamically animating the selected key from the virtual keyboard to indicate a remaining period of fixation that is required for the determination of the start of the eye-typing session.

[0021] In certain embodiments, the step of analysing the stream of gaze data results in determining a ranked list of possible words including the word, and wherein the presenting the determined word in the text presentation portion includes presenting and identifying a highest ranked possible word from the ranked list as the determined word and presenting a remaining portion of the ranked list as one or more alternative words.

[0022] In certain embodiments, after presentation of the ranked list, the method further includes:

detecting, based on the stream of gaze data, that the user’s gaze is fixated upon a selected alternative word from the one or more alternative words for a period of time greater than or equal to a word correction threshold; and

correcting the determined word to be the selected alternative word within the text presentation portion of the user interface.

[0023] In another aspect there is provided a processing system for enabling eye-typing, wherein the processing system is configured to:

receive a stream of gaze data including gaze samples directed toward a virtual keyboard of a user interface presented by a display device; determine that the user has eye-typed a word by detecting, based on the stream of gaze data, that the user’s gaze has moved to a text presentation portion of the user interface; in response to detecting that the user’s gaze is located within the text presentation portion of the user interface, analyse the stream of gaze data received prior to the detection to determine the word which the user has eye-typed;

present the determined word in the text presentation portion of the user interface; and determine that the user has begun eye-typing a further word based on a later portion of the stream of gaze data indicating that the user’s gaze has left the text presentation portion of the user interface.

[0024] In certain embodiments, the analysing the stream of gaze data includes:

determining, using the one or more keys, the word.

[0025] In certain embodiments, the processing system is configured to smooth, using a smoothing function, the plurality of gaze samples prior to determining the sequence of gaze velocities.

[0026] In certain embodiments, the smoothing function is a kernel smoother.

[0027] In certain embodiments, the processing system is configured to:

determine, based on the plurality of gaze samples, a sequence of vectors corresponding to a movement path of the user’s gaze;

determine, based on the sequence of vectors, a sequence of angular displacements between successive vectors from the sequence of vectors; and determine the word eye-typed by the user by searching a data store of words based on a search criteria indicative of the sequence of angular displacements, wherein each word in the data store has a respective sequence of angular displacements.

[0028] In certain embodiments, the processing system is configured to filter the sequence of angular displacements using an angular displacement threshold prior to the searching of the data store, wherein each angular displacement less than or equal to the angular displacement threshold is discarded.

[0029] In certain embodiments, the processing system is configured to determine a start of an eye -typing session based on detecting a user’s gaze being fixated upon a selected key from the virtual keyboard for a period of time that is greater than or equal to a dwell threshold, wherein the selected key represents a first keystroke that has been eye-typed by the user for the word.

[0030] In certain embodiments, prior to the determining the start of the eye-typing session and in response to determining that the user’s gaze is fixated upon the selected key from the virtual keyboard, the processing system is configured to dynamically animate the selected key from the virtual keyboard to indicate a remaining period of fixation that is required for the determination of the start of the eye-typing session.

[0031] In certain embodiments, the processing system is configured to:

analyse the stream of gaze data to determine a ranked list of possible words including the word; and

present the determined word in the text presentation portion by presenting and identifying a highest ranked possible word from the ranked list as the determined word and presenting a remaining portion of the ranked list as one or more alternative words.

[0032] In certain embodiments, after presentation of the ranked list, the processing system is configured to: detect, based on the stream of gaze data, that the user’s gaze is fixated upon a selected alternative word from the one or more alternative words for a period of time greater than or equal to a word correction threshold; and

correct the determined word to be the selected alternative word within the text presentation portion of the user interface.

[0033] In another aspect there is provided one or more computer readable mediums including executable instructions which, when executed by a processing system, configure the processing system to perform the method of the first aspect.

[0034] In another aspect there is provided a computer implemented method for launching a calibration session for an eye-typing system including the steps of:

displaying, by a display device, a user interface including a selectable termination control for cancelling a launch of the calibration session; and

subject to the termination control not having been selected, launching the calibration session after a predetermined time interval.

[0035] In certain embodiments, the termination control is selectable by a user’s gaze.

[0036] In certain embodiments, the method further includes the steps of:

receiving a stream of gaze data including gaze samples directed toward the user interface;

determining that the user has selected the termination control by detecting, based on the stream of gaze data, that the user’s gaze has moved to a portion of the user interface containing the termination control; and

subject to the termination control having been selected, cancelling the launch of the calibration session.

[0037] In certain embodiments, the method further includes the step of closing the user interface. [0038] In certain embodiments, the user interface further includes a timer display configured to count through the predetermined time interval.

[0039] In certain embodiments, the timer display is a countdown timer display configured to count downwards through the predetermined time interval.

[0040] In certain embodiments, the timer display is a count up timer display configured to count upwards through the predetermined time interval.

[0041] In certain embodiments, the predetermined time interval is five seconds.

[0042] In another aspect there is provided an eye-typing system including:

a display device; and

a processing system configured to:

display, with the display device, a user interface including a selectable termination control for cancelling a launch of a calibration session of the eye-typing system; and

subject to the termination control not having been selected, launch the calibration session after a predetermined time interval.

[0043] In certain embodiments, the eye-typing system further includes an eye-tracker, and wherein the termination control is selectable by a user’s gaze.

[0044] In certain embodiments, the processing system is further configured to:

receive, by the eye-tracker, a stream of gaze data including gaze samples directed toward the user interface;

determine that the user has selected the termination control by detecting, based on the stream of gaze data, that the user’s gaze has moved to a portion of the user interface containing the termination control; and subject to the termination control having been selected, cancel the launch of the calibration session.

[0045] In certain embodiments, the processing system is further configured to close the user interface.

[0046] In certain embodiments, the user interface further includes a timer display configured to count through the predetermined time interval.

[0047] In certain embodiments, the timer display is a countdown timer display configured to count downwards through the predetermined time interval.

[0048] In certain embodiments, the timer display is a count up timer display configured to count upwards through the predetermined time interval.

[0049] In certain embodiments, the predetermined time interval is five seconds

[0050] Other aspects and embodiments will be appreciated throughout the detailed description of the one or more preferred embodiments.

Brief Description of the Figures

[0051] Example embodiments should become apparent from the following description, which is given by way of example only, of at least one preferred but non-limiting embodiment, described in connection with the accompanying figures.

[0052] Figure 1 illustrates a functional block diagram of an example processing device that can be utilised to embody or give effect to a particular embodiment;

[0053] Figure 2 illustrates an example network infrastructure that can be utilised to embody or give effect to a particular embodiment;

[0054] Figure 3 is a flowchart representing an example method of enabling eye-typing; [0055] Figure 4 is an example user interface used in the enabling eye-typing;

[0056] Figure 5 is an example code snippet for determining a velocity trough of the gaze data;

[0057] Figure 6 is an example code snippet for smoothing gaze data;

[0058] Figure 7 is a plot of example gaze sample locations, key locations, and angular displacement samples;

[0059] Figure 8 is a plot of example gaze sample locations, key locations, and filtered angular displacement samples;

[0060] Figure 9 is an example code snippet for smoothing of gaze data to present a gaze indicator on the user interface;

[0061] Figure 10 is an example method for launching a calibration session for an eye-typing system;

[0062] Figure 11 is an example method for detecting selection of a termination control of a user interface by a user’s gaze;

[0063] Figure 12 is an example code snippet for launching a calibration session for an eye- typing system;

[0064] Figure 13 is an example display including an example user interface for launching a calibration session; and

[0065] Figure 14 is an example eye-typing system.

Detailed Description of Example Embodiments

[0066] The following modes, given by way of example only, are described in order to provide a more precise understanding of the subject matter of a preferred embodiment or embodiments. [0067] A particular embodiment of the present invention can be realised using a processing device, an example of which is shown in Fig. 1. In particular, the processing device 100 generally includes at least one processor 102, or processing unit or plurality of processors, memory 104, at least one input device 106 and at least one output device 108, coupled together via a bus or group of buses 110. In certain embodiments, input device 106 and output device 108 could be the same device. An interface 112 can also be provided for coupling the processing device 100 to one or more peripheral devices, for example interface 112 could be a PCI card or PC card. At least one storage device 114 which houses at least one database 116 can also be provided. The memory 104 can be any form of memory device, for example, volatile or non-volatile memory, solid state storage devices, magnetic devices, etc.

[0068] The processor 102 could include more than one distinct processing device, for example to handle different functions within the processing device 100.

[0069] Input device 106 receives input data 118 (such as electronic content data), for example via a network or from a local storage device. Output device 108 produces or generates output data 120 (such as viewable content) and can include, for example, a display device or monitor in which case output data 120 is visual, a printer in which case output data 120 is printed, a port for example a USB port, a peripheral component adaptor, a data transmitter or antenna such as a modem or wireless network adaptor, etc. Output data 120 could be distinct and derived from different output devices, for example a visual display on a monitor in conjunction with data transmitted to a network. A user could view data output, or an interpretation of the data output, on, for example, a monitor or using a printer. The storage device 114 can be any form of data or information storage means, for example, volatile or non-volatile memory, solid state storage devices, magnetic devices, etc..

[0070] Examples of electronic data storage devices 114 can include disk storage, optical discs, such as CD, DVD, Blu-ray Disc, flash memory/memory card (e.g., solid state semiconductor memory), MultiMedia Card, USB sticks or keys, flash drives, Secure Digital (SD) cards, microSD cards, miniSD cards, SDHC cards, miniSDSC cards, solid-state drives, and the like. [0071] In use, the processing device 100 is adapted to allow data or information to be stored in and/or retrieved from, via wired or wireless communication means, the at least one database 116. The interface 112 may allow wired and/or wireless communication between the processing unit 102 and peripheral components that may serve a specialised purpose. The processor 102 receives instructions as input data 118 via input device 106 and can display processed results or other output to a user by utilising output device 108. More than one input device 106 and/or output device 108 can be provided. It should be appreciated that the processing device 100 may be any form of terminal, PC, laptop, notebook, tablet, smart phone, specialised hardware, or the like.

[0072] The processing device 100 may be a part of a networked communications system 200, as shown in Fig. 2. Processing device 100 could connect to network 202, for example the Internet or a WAN. Input data 118 and output data 120 could be communicated to other devices via network 202. Other terminals, for example, thin client 204, further processing systems 206 and 208, notebook computer 210, mainframe computer 212, PDA 214, pen- based computer 216, server 218, etc., can be connected to network 202. A large variety of other types of terminals or configurations could be utilised. The transfer of information and/or data over network 202 can be achieved using wired communications means 220 or wireless communications means 222. Server 218 can facilitate the transfer of data between network 202 and one or more databases 224. Server 218 and one or more databases 224 provide an example of an information source.

[0073] Other networks may communicate with network 202. For example, telecommunications network 230 could facilitate the transfer of data between network 202 and mobile or cellular telephone 232 or a PDA-type device 234, by utilising wireless communication means 236 and receiving/transmitting station 238. Satellite communications network 240 could communicate with satellite signal receiver 242 which receives data signals from satellite 244 which in turn is in remote communication with satellite signal transmitter 246. Terminals, for example further processing system 248, notebook computer 250 or satellite telephone 252, can thereby communicate with network 202. A local network 260, which for example may be a private network, LAN, etc., may also be connected to network 202. For example, network 202 could be connected with Ethernet 262 which connects terminals 264, server 266 which controls the transfer of data to and/or from database 268, and printer 270. Various other types of networks could be utilised.

[0074] The processing device 100 is adapted to communicate with other terminals, for example further processing systems 206, 208, by sending and receiving data, 118, 120, to and from the network 202, thereby facilitating possible communication with other components of the networked communications system 200.

[0075] Thus, for example, the networks 202, 230, 240 may form part of, or be connected to, the Internet, in which case, the terminals 206, 212, 218, for example, may be web servers, Internet terminals or the like. The networks 202, 230, 240, 260 may be or form part of other communication networks, such as LAN, WAN, Ethernet, token ring, FDDI ring, star, etc., networks, or mobile telephone networks, such as GSM, CDMA or 3G, etc., networks, and may be wholly or partially wired, including for example optical fiber, or wireless networks, depending on a particular implementation.

Example 1: Eye-typing method and system

[0076] Referring to Figure 3 there is shown a flowchart representing a method 300 for enabling eye-typing. The method 300 can be performed by a processing system such as processing system 100. Furthermore, the processing system 100 can be configured by one or more computer readable mediums which include executable instructions to perform the method 300.

[0077] In particular, as step 310 the method 300 includes receiving a stream of gaze data including gaze samples directed toward a virtual keyboard of a user interface presented by a display device.

[0078] At step 320, the method 300 includes determining that the user has eye-typed a word by detecting that, based on the stream of gaze data, the user’s gaze has moved to a text presentation portion of the user interface. [0079] At step 330, in response to detecting that the user’s gaze is located within the text presentation portion of the user interface, the method 300 includes analysing the stream of gaze data received prior to the detection to determine the word which the user has eye-typed.

[0080] At step 340, the method 300 includes presenting the determined word in the text presentation portion of the user interface.

[0081] At step 350, the method 300 includes determining that the user has begun eye-typing a further word based on a later portion of the stream of gaze data indicating that the user’s gaze has left the text presentation portion of the user interface.

[0082] Prior art systems require the user to select one or more keys to indicate an end of the word or the start of word which can slow the user’s eye-typing speed. Other prior art systems involve performing a particular eye action such as blinking which can be difficult to learn and also can slow the user’s eye-typing speed. Advantageously, due to an observed behavior where the user naturally wishes to check they have entered each word correctly in the text presentation portion of the user interface before moving to the next word, interpreting the movement of the user’ s gaze to and from the text presentation portion as the end of the word and start of a further word respectively enables substantial increases in eye-typing speed because the user does not need to undertake additional actions to provide this input.

[0083] It will be appreciated that steps 320 to 350 of method 300 can be repeated for the further word and subsequent further words.

[0084] Referring to Figure 4 there is shown an example of the user interface 400 including the virtual keyboard 410 and the text presentation portion 420. In this particular example, the text presentation portion 420 is located above the virtual keyboard 410. Furthermore, in this particular example, the virtual keyboard 410 is a QWERTY keyboard. However, it will be appreciated that other keyboard layouts can alternatively be used. As discussed in further detail below, the user interface can also include a plurality of alternate intended words as the result of analyzing the gaze data. [0085] Additional increases in eye-typing speed can be achieved based on a velocity of movement of the user’s gaze. A number of prior art systems that use dwell time effectively discard gaze samples that are not clustered upon a key location of the virtual keyboard. However, the Applicant has appreciated that the gaze samples obtained whilst the user’s gaze moves between keys can be highly informative of the particular key which the user is wishing to select. In particular, the processing system can be configured to analyse the stream of gaze data by firstly determining, based on the gaze samples obtained at a predefined sampling frequency, a sequence of gaze velocities. Next, the method includes determining, based on the sequence of gaze velocities, one or more keys of the virtual keyboard corresponding to a respective one or more gaze velocity troughs. Each gaze velocity trough can be a minima in gaze velocity, wherein each gaze trough is located between a decrease and an increase in the rate of velocity. The processing system 100 is configured to determine, using the one or more keys corresponding to the one or more gaze velocity troughs, the word. An example code snippet for determining the one or more gaze velocity troughs is provided in Figure 5.

[0086] Prior to the determining the sequence of gaze velocities, the method includes smoothing, using a smoothing function, the plurality of gaze samples. The smoothing function can be a kernel smoother such as Gaussian Kernel Regression. In particular, to utilise Gaussian Kernel Regression for smoothing a data set, the gaussian kernel function is used as a weight function in the standard weighted average equation. The Weighted Average function is shown in Equation 1 and the Kernel Function is shown in Equation 2. The combined form of these equations is shown in Equation 3 below, where a smoothed point can be determined by providing the test point, the rest of the data set for weight determination and the sigma value, which represents the smoothing value.

Equation 1

Equation 2

[0087] An implementation of this method of smoothing data points is provided in the example code snippet provided in Figure 6 where two core functions are defined, one for obtaining the weights given a point and a dataset and the other is for smoothing a data point based on supplied weights and data.

[0088] Additional increases in eye-typing speed can also be achieved by using the path of the user’s gaze as search criteria for searching a data store of words. For example this can include performing pattern recognition in relation to a dictionary of words which are indexed according to a respective gaze path. In one particular implementation, the processing system 100 can be configured to determine, based on the plurality of gaze samples, a sequence of vectors corresponding to a movement path of the user’s gaze. An example of the vectors determined for recorded gaze samples of the eye-typed word“hello” are shown in Figure 7. The processing system 100 then determines, based on the sequence of vectors, a sequence of angular displacements between successive vectors from the sequence of vectors. Finally, the processing system 100 is configured to determine the word eye-typed by the user by searching a data store of words based on a search criteria indicative of the sequence of angular displacements, wherein each word in the data store has a respective sequence of angular displacements. In one form, the search criteria may be a hash of the sequence of the angular displacements. In this arrangement, each word in the data store can include a unique hash of the corresponding sequence of the angular displacements of the respective word.

[0089] More specifically, a grid-based technique is used to determine the vectors, where vectors are sampled in a dynamic grid and the angular difference between vectors is calculated. The grid is circular and an arbitrary radius is chosen beforehand. The moment in which a gaze point is detected at a distance larger than the arbitrary radius from the origin, the vector is finalised and a new origin is defined for the next vector. The initial vector’s origin is selected as the starting point of the gaze data and once a gaze sample is detected an arbitrary distance from the origin, it defines the destination of the previous vector and the new origin of the next vector. On the detection of origin of the third vector and every new vector continuing, angles are sampled. It’s important to note that the angles displayed in Figure 7 are shown next to the gaze sample at the time in which they are detected, i.e. the destination of the second vector in the angle calculation, not the point at which both vectors meet that creates the angle shown.

[0090] Due to the high frequency that gaze samples can be collected, an angular displacement between successive vectors in the sequence of vectors may show very small angular displacements which can result in various problems. As such, prior to searching the data store, the method can include filtering the sequence of angular displacements using an angular displacement threshold. Each angular displacement less than or equal to the angular displacement threshold can be discarded. In one example, the angular displacement threshold is set to 30 degrees. An example plot of the filtered angular difference samples determined for the eye-typed word“hello” is shown in Figure 8.

[0091] Additional increases in eye-typing speed can also be achieved through the use of auto-suggest functionality, wherein a plurality of potential words can be presented to the user within the user interface. In particular, the method includes analysing the stream of gaze data which results in determining a ranked list of possible intended words including the word. The method then includes presenting a highest ranked possible word from the ranked list as the determined word in the text presentation portion 420 of the user interface and presenting a remaining portion of the ranked list as one or more alternative words in alternate word suggestion fields 430 of the user interface 400. In the event that the processing system 100 has been able to correctly determine the eye-typed word as displayed in the text presentation portion of the user interface, then the user can simply move their gaze back to the virtual keyboard to begin eye-typing the next word. However, in the event that user is of the opinion that the processing system 100 has been unable to correctly determine the eye-typed word but one of the alternative words presented within the user interface is the intended word, the method can include detecting, based on the stream of gaze data, that the user’s gaze is fixated upon a selected alternative word from the one or more alternative words for a period of time greater than or equal to a word correction threshold. The method can then include correcting the determined word to be the selected alternative word within the text presentation portion 420 of the user interface 400.

[0092] Additional increases in eye-typing speed can also be achieved by interpreting a single action for dual purposes. For example, the method can include determining a start of an eye -typing session based on detecting a user’s gaze being fixated upon a selected key from the virtual keyboard for a period of time that is greater than or equal to a dwell threshold, wherein the selected key represents a first keystroke that has been eye-typed by the user for the word. Thus, in this instance the fixation of the user’s gaze on the selected key has the dual purpose of indicating the start of the eye-typing session and also selecting the first letter of the word to be eye-typed. In certain examples, the method can include dynamically animating the selected key from the virtual keyboard during the time period that the user’s gaze has been detected as being fixed on the selected key. The dynamic animation provides feedback as to the length of time remaining before the start of the eye-typing session has been detected. For example, the dynamic animation may be a circle with a dynamically filled segment indicating a percentage of the dwell threshold period that has been completed by the user’s fixation gaze (e.g. a third of the circle is filled by the segment when a third of the dwell threshold period has been detected).

[0093] It will be appreciated that whilst the processing system 100 can include an in-built camera system to track the user’s gaze, it is also possible for the processing system 100 to be coupled to a peripheral device via an interface to track the user’s eyes. For example, suitable devices include a Tobii peripheral, such as a Tobii EyeMobile Mini eye tracking device or the Tobii 4C eye tracking device, can be used.

[0094] Whilst the processing system could potentially be a PC, laptop, or the like, it is possible that the processing system could be part of a Virtual Reality, Mixed Reality or Augmented Reality headset.

[0095] In one form, the data store of words could be provided in the form of a database. In one form, the database may be managed by a database engine preferably provided in the form of a self-contained, server-less, zero-configuration, transactional SQL database engine. An example of such a database engine is SQLite.

[0096] In one form, the processing system can be configured to determine, using the stream of gaze data, a location on the user interface which the user is gazing and present an indicator 440 on the user interface to show the determined location. In one example, the indicator may be a circular marker as shown in Figure 4. In preferable embodiments, the gaze samples of the gaze data may be smoothed prior to presentation of the gaze indicator on the user interface. In one form, a sliding average smoothing technique can be used. Alternatively, an exponential smoothing function can be used to smooth the gaze samples. The exponential smoothing function can use non-uniform weighting where the weighting is exponentially distributed so that each point further away from the current point is weighted less than the last. The exponential smoothing function can use a smoothing factor which can be tuned. In experiments, it has been found advantageous to use different smoothing factors when the user’s gaze is stationary versus when the user’s gaze is moving over the user interface 400. As a result, it has been found preferable that two arbitrary smoothing factors be defined, one for each state. When the gaze data is detected as being in one of these states, the smoothing factor is set to the corresponding smoothing factor for that state. To use an arbitrary threshold for toggling the smoothing mode between fixation/mid-fixation, it is suitable to smooth the velocity data first, with its own arbitrary smoothing factor. As a result, two exponential smoothing algorithm implementations can be used successively, wherein the first is used to calculate the smoothing factor input of the second. An example code snippet implementing this method is provided in Figure 9.

Example 2: Calibration session launcher

[0097] Referring to Figure 10, there is shown an example method 500 for launching a calibration session for an eye-typing system.

[0098] Method 500 includes step 510 of displaying a user interface including a selectable termination control for cancelling, or aborting, the launch of the calibration session. The user interface is presented or displayed by a display device, such as an electronic visual display screen or a projector. Other examples of the display device include visual display screens for virtual reality or augmented reality headsets, head-mounted displays, stereoscopic displays, or any other type of device or system configured to display one or more images or videos.

[0099] Method 500 further includes step 520, subject to the termination control not having been selected, of launching the calibration session after a predetermined time interval, or a predetermined time period. In some examples, the calibration session is launched automatically after the predetermined time period, without any further input or action from the user. Therefore, method 500 may provide a method for automatic eye -position calibration.

[00100] Method 500 may be initiated or executed automatically by a processing system prior to launching a calibration session. This may occur at any stage, such as at start-up or during operation of the processing system, for example, when the processing system is executing or running a primary program or application. In some examples, following execution of method 500 and of the calibration session, the processing system may proceed to execute an eye-typing session, such as an eye-typing session using method 300 for enabling eye-typing.

[00101] The processing system may be programmed or configured such that method 500 is a preamble to a calibration session. Alternatively, the processing system may be programmed or configured such that method 500 interrupts a normal execution of the calibration session. Therefore, in some examples, method 500 further includes a step of detecting a launch of the calibration session, and a step of suspending the launch of the calibration session, for example, by informing the processor that a higher-priority activity needs to precede the launch of the calibration session. Then, the processing system may begin executing steps 510 and 520.

[00102] Step 520 of launching the calibration session may further include closing the user interface (either before or immediately after launching of the calibration session), such that the user interface does not overlay the calibration session images, or otherwise disturb the user during the calibration session. [00103] The calibration session may be any type of calibration session or routine for eye typing systems, eye-tracking systems, eye-blink sensors, switch operation, muscle control, electromyography (EMG) control, electrooculography (EOG) control, electroencephalography (EEG) control, brain-computer interfaces, or any other type of system, sensor, or interface that may require calibration. For example, the calibration session may calibrate the eye-typing system to characteristics of the gaze of a specific user.

[00104] The predetermined time interval may be predetermined by the user, by a technician, or by the eye-typing system itself. For example, the predetermined time interval may be provided as part of a setting of the eye-typing system.

[00105] The predetermined time interval should be sufficiently long to allow a user to decide whether they wish to proceed with calibration of the eye-typing system. However, the predetermined time interval should not be excessively long to avoid inconvenient wait times before the calibration session is launched. In some examples, the predetermined time interval should not exceed an average duration of the calibration session.

[00106] In some examples, the predetermined time interval is five seconds. In other examples, the predetermined time interval may be any length of time, such as one, two, three, four, five or more seconds. Preferably, though not necessarily, the predetermined time interval is a seconds-long time interval (i.e. less than one minute long).

[00107] In some examples, the predetermined time interval is a time interval, a time period, or a wait time, measured from the moment when the user interface is displayed (i.e. from the moment of execution of step 510). In other examples, the predetermined time interval may be defined with reference to any other moment or event.

[00108] The termination control is a selectable control of the user interface, such as a virtual push-button, a virtual switch, selectable text, or any other virtual control having two states. In a first state,“unselected” state, or default state of the termination control, step 520 is executed to launch the calibration session. In a second state, or“selected” state, of the termination control, step 520 is not executed, preventing the launch of the calibration session. [00109] Preferably, though not necessarily, the termination control is selectable by a user’s gaze, or other eye-control movement (e.g. eye blinks). Therefore, method 500 may include additional steps, shown in Figure 11, to detect selection of the termination control by the user’s gaze.

[00110] Method 500 optionally includes step 512 of receiving a stream of gaze data including gaze samples directed toward the user interface.

[00111] Method 500 further optionally includes step 514 of determining that the user has selected the termination control by detecting, based on the stream of gaze data, that the user’s gaze has moved to the termination control, or to a vicinity of the termination control in a portion of the user interface containing the termination control.

[00112] Method 500 further optionally includes step 516, subject to the termination control having been selected, of cancelling the launch of the calibration session.

[00113] Steps 512, 514, and 516 may be executed after step 510 but before step 520. That is, steps to detect selection of the termination control by the user’s gaze are performed after displaying the user interface, but before launching the calibration session.

[00114] In some examples, the termination control is selectable by manual command, such as by pressing a button on a controller or by using a mouse.

[00115] In some examples, method 500 further includes the step of closing the user interface. The user interface may be closed either after launching of the calibration session or after cancelling the launch of the calibration session. The step of closing the user interface includes stopping the displaying of the user interface by the display device.

[00116] In some examples, the user interface is displayed until the termination control is selected. In other examples, the user interface is displayed until the calibration session is launched.

[00117] Following the closing of the user interface, in the event that the termination control has been selected, the processing system may return to executing or running a primary program or application that was being run prior to the initiation of method 500. Alternatively, in the event that the termination control has not been selected, the processing system launches the calibration session.

[00118] Advantageously, method 500 provides a way for a user to cancel a calibration session or routine called automatically by an eye-typing system. When no calibration is needed, the user is able to select the termination control with their gaze, thus preventing the launch of the calibration session. Alternatively, when calibration is needed, the user can simply wait until the end of the predetermined time interval for the calibration session to launch.

[00119] Referring to Figure 12, there is shown an example code snippet for launching a calibration session for an eye-typing system. The code snippet provides a first function, “StartO”, to begin a timer. The code snippet further includes a second function,“Update()”, to update a time-counter variable and to launch the calibration session when the time-counter variable equals or exceeds a predetermined time interval. The code snippet further includes a third function,“CancelCalibration()”, to stop the count down and to prevent a launch of the calibration session.

[00120] Preferably, though not necessarily, the user interface is a graphical user interface that is displayed in a virtual window or tab of the virtual display device. Preferably, though not necessarily, the user interface overlays or is superimposed on other windows, tabs, or pictures being displayed by the display device. In some examples, the user interface is an on screen display.

[00121] In some examples, the user interface further includes a timer display configured to count through the predetermined time interval. The timer display informs the user of how much time they have to wait before the launch of the of the calibration session.

[00122] The predetermined time interval may be defined as an interval in time between a lower time point, or limit (e.g. t = 0 seconds) and an upper time point, or limit (e.g. t = 5 seconds). The difference in time between the upper time limit and the lower time limit equals the predetermined time interval. [00123] In some examples, the timer display is a countdown timer display configured to count downwards through the predetermined time interval. That is, the timer display counts downwards from the upper time point to the lower time point. In some examples, the countdown timer display counts down in decrements of one second.

[00124] In some examples, the timer display is a count up timer display configured to count upwards through the predetermined time interval. That is, the timer display counts upwards from the lower time point to the upper time point. In some examples, the count up timer display counts up in increments of one second.

[00125] Referring to Figure 13, there is shown an example display 600 of the display device. Display 600 includes a user interface 610 for launching a calibration session. User interface 610 overlays, or is superimposed onto, a deactivated user interface 620 of a primary program or application being executed by the processing system. Primary user interface 620 is shaded or blackened to inform the user that the primary program has been deactivated whilst user interface 610 is active.

[00126] User interface 610 includes a selectable termination control 612 for cancelling the launch of the calibration session. User interface 610 further includes a countdown timer display 614 for counting down the predetermined time interval.

[00127] Figure 13 also shows a gaze locator, or indicator, 630 locating the user’s gaze on the display device. The user can cancel the launch of the calibration session by moving their gaze such that gaze locator 630 overlaps termination control 612, thus selecting termination control 612. Alternatively, the user can proceed with launching the calibration session by moving their gaze such that gaze locator 630 does not overlap, and does not select, termination control 612, and by waiting until countdown timer display 614 has completed the count down of the predetermined time interval.

[00128] Method 500 can be performed by a processing system such as processing system 100. Furthermore, processing system 500 can be configured by one or more computer readable mediums which include executable instructions to perform method 500. [00129] Referring to Figure 14, there is shown an example eye-typing system 700. Eye typing system 700 includes a display device 710, and a processing system 720. Processing system 720 may be the same as processing system 100.

[00130] Processing system 720 is configured to display, with display device 710, a user interface including a selectable termination control for cancelling, or aborting, a launch of a calibration session of eye-typing system 700.

[00131] Display device 710 may be an electronic visual display screen or a projector. Other examples of display device 710 include visual display screens for virtual reality or augmented reality headsets, head-mounted displays, stereoscopic displays, or any other type of device or system configured to display one or more images or videos.

[00132] Processing system 720 is further configured, subject to the termination control not having been selected, to launch the calibration session after a predetermined time interval.

[00133] Eye-typing system 700 further includes a sensor 730, which may be an eye- tracker, or eye-tracking system for tracking the position on the screen on which a user's eyes are focused. In some examples, the calibration session is configured to calibrate sensor 730.

[00134] Processing system 720 is connected to display device 710 and to sensor 730. Furthermore, display device 710 is connected to sensor 730. In some examples, sensor 730 is mounted onto, or on top of, display device 710. In some examples, display device 710, processing system 720, and sensor 730 are integrally formed as part of a unitary eye-tracking system 700, for example, as part of a tablet or a laptop computer.

[00135] Preferably, though not necessarily, the termination control is selectable by a user’s gaze. In some examples, processing system 720 is further configured to receive, by sensor 730, a stream of gaze data including gaze samples directed toward the user interface. In some examples, processing system 720 is further configured to determine that the user has selected the termination control by detecting, based on the stream of gaze data, that the user’s gaze has moved to a portion of the user interface containing the termination control. In some examples, processing system 720 is further configured, subject to the termination control having been selected, to cancel the launch of the calibration session.

[00136] In some examples, processing system 720 is further configured to close the user interface.

[00137] More generally, system 700 may be an eye-typing system, eye-tracking system, eye -blink sensor system, switch operation system, muscle control system, electromyography (EMG) control system, electrooculography (EOG) control system, electroencephalography (EEG) control system, brain-computer interface system, or any other type of system, sensor, or interface that may require calibration.

[00138] Many modifications within the scope of the invention will be appreciated by those skilled in the art without department from the spirit of the invention.

[00139] Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

Claims

1. A computer implemented method for enabling eye-typing including steps of:

presenting the determined word in the text presentation portion of the user interface; and

determining that the user has begun eye-typing a further word based on a later portion of the stream of gaze data indicating that the user’s gaze has left the text presentation portion of the user interface.

2. The computer implemented method of claim 1, wherein the step of analysing the stream of gaze data includes:

determining, using the one or more keys, the word.

3. The computer implemented method of claim 2, wherein prior to the determining the sequence of gaze velocities, the method includes smoothing, using a smoothing function, the plurality of gaze samples.

4. The computer implemented method of claim 3, wherein the smoothing function is a kernel smoother.

5. The computer implemented method of any one of claims 2 to 4, wherein the method includes:

determining, based on the plurality of gaze samples, a sequence of vectors corresponding to a movement path of the user’ s gaze;

6. The computer implemented method of claim 5, wherein prior to the search of the data store, the method includes filtering the sequence of angular displacements using an angular displacement threshold, wherein each angular displacement less than or equal to the angular displacement threshold is discarded and one or more subsequent angular displacements are adjusted according to the discarded angular displacement.

7. The computer implemented method of any one of claims 1 to 6, wherein the method includes determining a start of an eye-typing session based on detecting a user’s gaze being fixated upon a selected key from the virtual keyboard for a period of time that is greater than or equal to a dwell threshold, wherein the selected key represents a first keystroke that has been eye-typed by the user for the word.

8. The computer implemented method of claim 7, wherein prior to the determining the start of the eye-typing session and in response to determining that the user’s gaze is fixated upon the selected key from the virtual keyboard, the method includes dynamically animating the selected key from the virtual keyboard to indicate a remaining period of fixation that is required for the determination of the start of the eye-typing session.

9. The computer implemented method of any one of claims 1 to 8, wherein the step of analysing the stream of gaze data results in determining a ranked list of possible words including the word, and wherein the presenting the determined word in the text presentation portion includes presenting and identifying a highest ranked possible word from the ranked list as the determined word and presenting a remaining portion of the ranked list as one or more alternative words.

10. The computer implemented method of claim 9, wherein after presentation of the ranked list, the method further includes:

11. A processing system for enabling eye-typing, wherein the processing system is configured to:

receive a stream of gaze data including gaze samples directed toward a virtual keyboard of a user interface presented by a display device;

determine that the user has eye-typed a word by detecting, based on the stream of gaze data, that the user’s gaze has moved to a text presentation portion of the user interface; in response to detecting that the user’s gaze is located within the text presentation portion of the user interface, analyse the stream of gaze data received prior to the detection to determine the word which the user has eye-typed;

12. The processing system of claim 11, wherein the analysing the stream of gaze data includes:

determining, using the one or more keys, the word.

13. The processing system of claim 12, wherein the processing system is configured to smooth, using a smoothing function, the plurality of gaze samples prior to determining the sequence of gaze velocities.

14. The processing system of claim 13, wherein the smoothing function is a kernel smoother.

15. The processing system of any one of claims 12 to 14, wherein the processing system is configured to:

determine, based on the sequence of vectors, a sequence of angular displacements between successive vectors from the sequence of vectors; and

determine the word eye-typed by the user by searching a data store of words based on a search criteria indicative of the sequence of angular displacements, wherein each word in the data store has a respective sequence of angular displacements.

16. The processing system of claim 15, wherein the processing system is configured to filter the sequence of angular displacements using an angular displacement threshold prior to the searching of the data store, wherein each angular displacement less than or equal to the angular displacement threshold is discarded.

17. The processing system of any one of claims 11 to 16, wherein the processing system is configured to determine a start of an eye-typing session based on detecting a user’s gaze being fixated upon a selected key from the virtual keyboard for a period of time that is greater than or equal to a dwell threshold, wherein the selected key represents a first keystroke that has been eye-typed by the user for the word.

18. The processing system of claim 17, wherein prior to the determining the start of the eye-typing session and in response to determining that the user’s gaze is fixated upon the selected key from the virtual keyboard, the processing system is configured to dynamically animate the selected key from the virtual keyboard to indicate a remaining period of fixation that is required for the determination of the start of the eye-typing session.

19. The processing system of any one of claims 11 to 18, wherein processing system is configured to:

20. The processing system of claim 19, wherein after presentation of the ranked list, the processing system is configured to:

detect, based on the stream of gaze data, that the user’s gaze is fixated upon a selected alternative word from the one or more alternative words for a period of time greater than or equal to a word correction threshold; and

21. One or more computer readable mediums including executable instructions which, when executed by a processing system, configure the processing system to perform the method of any one of any one of claims 1 to 10.

22. A computer implemented method for launching a calibration session for an eye-typing system including the steps of:

23. The computer implemented method of claim 22, wherein the termination control is selectable by a user’s gaze.

24. The computer implemented method of claim 23, wherein the method further includes the steps of:

25. The computer implemented method of any one of claims 22 to 24, wherein the method further includes the step of closing the user interface.

26. The computer implemented method of any one of claims 22 to 25, wherein the user interface further includes a timer display configured to count through the predetermined time interval.

27. The computer implemented method of claim 26, wherein the timer display is a countdown timer display configured to count downwards through the predetermined time interval.

28. The computer implemented method of claim 26, wherein the timer display is a count up timer display configured to count upwards through the predetermined time interval.

29. The computer implemented method of any one of claims 22 to 28, wherein the predetermined time interval is five seconds.

30. An eye-typing system including:

a display device; and

a processing system configured to:

31. The eye-typing system of claim 30, wherein the eye-typing system further includes an eye-tracker, and wherein the termination control is selectable by a user’s gaze.

32. The eye-typing system of claim 31, wherein the processing system is further configured to:

32. The eye-typing system of any one of claims 28 to 30, wherein the processing system is further configured to close the user interface.

33. The eye-typing system of any one of claims 30 to 32, wherein the user interface further includes a timer display configured to count through the predetermined time interval.

34. The eye-typing system of claim 33, wherein the timer display is a countdown timer display configured to count downwards through the predetermined time interval.

35. The eye-typing system of claim 33, wherein the timer display is a count up timer display configured to count upwards through the predetermined time interval.

36. The eye-typing system of any one of claims 30 to 35, wherein the predetermined time interval is five seconds.