EP2828725A1

EP2828725A1 - User input system

Info

Publication number: EP2828725A1
Application number: EP13710528.4A
Authority: EP
Inventors: Tom KAVLI; Hans Joergen Bang
Original assignee: Elliptic Laboratories ASA
Current assignee: Elliptic Laboratories ASA
Priority date: 2012-03-05
Filing date: 2013-03-05
Publication date: 2015-01-28
Also published as: WO2013132244A1; GB201203832D0

Abstract

A user input to a system is processed by transmitting one or more signals through air towards a moving input object (12). A first signal, comprising a reflection from the input object (12), is received during a first time period, and is used to determine a characteristic of the movement of the input object (12). A second signal, comprising a further reflection from the input object (12), is received during a second time period, starting after the first time period has ended. The second received signal and the movement characteristic are together used to determine an input, to which the system responds.

Description

User Input System

This invention relates to methods and systems for processing a user input made using a moving input object; in particular, but not exclusively, for processing touchless, gesture-based inputs to electronic devices.

It is known for a computing device to support touchless interaction by providing one or more ultrasonic transmitters and receivers on the device and determining movement of an input object, such as a user's hand, by analysing reflections of ultrasonic signals off the object. For example, WO 2009/1 15799, by the present applicant, describes a system for recognising ultrasound-sound-based gesture inputs.

However, touchless gesture input systems can sometimes be unreliable. In particular, a user's intended input may be ignored completely by the system (a missed detection) or it may mistakenly be recognised as a different input from that which the user intended (an incorrect detection). Alternatively, a casual movement by a user may mistakenly be detected as an input to the system when none way intended (a false positive). In each of these scenarios, the system does not respond as desired. This can lead to user frustration.

The present invention seeks to provide motion-based input systems with improved accuracy. From a first aspect, the invention provides a method of processing a user input to a system, comprising:

transmitting one or more signals through air towards a moving input object; receiving, during a first time period, a first signal comprising a reflection from the input object;

using the first signal to determine a characteristic of the movement of the input object;

receiving, during a second time period, starting after the first time period has ended, a second signal comprising a further reflection from the input object; using (i) the second received signal, and (ii) said characteristic, to determine input to the system; and

responding to the input. From another aspect, the invention provides a system for processing a user input, comprising receiving means configured to receive signals through air, and processing means configured to:

process a first signal, received during a first time period, comprising a reflection of a transmitted signal through air from a moving input object, so as to determine a characteristic of the movement of the input object;

process (i) a second signal, received during a second time period, starting after the first time period has ended, comprising a further reflection from the input object, and (ii) said characteristic, so as to determine an input to the system; and respond to the input.

The invention extends to a software product, and to a carrier or signal bearing the same, comprising instructions which, when executed by processing means, cause the processing means to:

process a first signal, received by receiving means during a first time period, comprising a reflection of a transmitted signal through air from a moving input object, so as to determine a characteristic of the movement of the input object; process (i) a second signal, received during a second time period, starting after the first time period has ended, comprising a further reflection from the input object, and (ii) said characteristic, so as to determine an input; and

respond to the input.

Thus, it will be seen by those skilled in the art that, in accordance with the invention, information about the input object's movement during a first time period can be used to help interpret its subsequent movement during a later time period. In this way, the system can "learn" from historic input movements in order to improve the accuracy of future input determinations. The nature of this learning may take many different forms.

Although not essential, the system may also process the first signal to determine an (earlier) input to the system, and may additionally respond to that input. The movement of the input object during the first time period may directly determine such an earlier input to the system (i.e. without involving the second signal), to which the system responds (for example, the movement may be a horizontal sweep gesture, which causes a web browser application to go back to the previous page). The input determined from the movement during the second time period may then be a separate input (e.g. an instruction to minimise an open window on a display screen).

Alternatively, movements of the input object during the first and second time periods may, together, determine an input to the system; for example, a circling gesture may be commenced in the first time period and continued during the second time period, which in its entirety causes the system to invoke a

predetermined function. Movement of the input object in the first time period may be used by the system to improve the accuracy of the input recognition due to the signal received during the second time period.

An input may be a discrete event, e.g. an instruction which is identified and executed only once a predetermined movement or gesture of the input object has been completed, or it may be an on-going or continual event or series of events, to which the system responds in a continuous or continuing manner (e.g. continuously rotating a virtual scroll wheel or slider to cause a continuous or incrementally ongoing response by the device). It will be appreciated that references herein to continuous or real-time responses or processes may, in practice, relate to approximations thereto, implemented as a rapid succession of discrete responses or processes.

The characteristic of the movement of the input object may be determined from data representing the position of the input object at two or more moments during the first time period. The processing means may, for example, determine the characteristic by processing a set or series of coordinates in two or preferably three dimensions, where each coordinate represents the location of the input object (or an estimate thereof) at a different moment in time. These moments may be spaced at intervals, preferably regular intervals, across the first time period. The characteristic may, however, be determined by any suitable processing of the first signal. For example, the processing means may apply one or more filters to a representation of the first signal, e.g. using an approach as described in WO 2009/115799, the contents of which are hereby incorporated in their entirety, without necessarily determining a succession of position coordinates. It may thereby determine a characteristic such as the general direction or speed of movement of the input object during the first time period.

However it is determined, the characteristic of the movement of the input object may comprise any suitable geometric or statistical property of the object's motion during the first time period (and optionally also during other time periods). In particular, it may comprise one or more of: the speed or velocity of the object in three dimensional space; a component of the speed or velocity of the object in a particular direction, plane or surface; the direction of movement of the object; a plane within which, or parallel to which, the movement occurs; and the extent or scale of the object's movement over a time window. For each of these cases, the characteristic may represent a principal, average (e.g. mean), minimum or maximum value over a time window, such as the first time period. Alternatively, it may comprise a plurality of data points, such as a series of such values determined at intervals.

The system may receive additional signals comprising further reflections from the input object during the first time period; these may be used in addition to the first signal to determine the characteristic of the movement of the input object. In some embodiments, the second signal may be used in addition to the first signal to determine the characteristic of the movement of the input object (e.g. the same position data may be used first to estimate certain features or characteristics of the object's movement, and then used again to decide on the input based on that movement). However, in preferred embodiments, the second signal is not used for determining the characteristic; the characteristic preferably relates only to movement of the input object occurring before the start of the second time period.

The system may receive additional signals comprising further reflections from the input object during the second time period, and may use these in addition to the second received signal to determine the input to the system. The first and/or second signals may each comprise a plurality of reflections, e.g. reflections from different parts of the input object and/or reflections of multiple transmitted signals separated in time, frequency, direction, coding, etc.

The input object is preferably in motion during at least some of the second time period. In some embodiments, the system is configured to estimate a trajectory of the input object from the signal received during the second time period, and to use this trajectory when determining the input. The characteristic of the movement of the input object during the first time window may be used to modify the estimated trajectory; for instance, to increase the accuracy of this estimated trajectory. This may be such that the estimate is closer to the true trajectory of the input object, or such that it is closer to an idealised trajectory for an intended input. The modifying may comprise the use of Bayesian inference. In some embodiments, the first signal may be used to determine a plane (e.g. as described below); the second signal may be used to estimate a three-dimensional trajectory relating to movement of the input object during the second time window; and this trajectory may then be project onto the plane. More generally, in one set of embodiments, the determined characteristic of the movement of the input object is, or comprises, a plane within which, or parallel to which, the movement principally or substantially occurs. Such a plane may conveniently be represented by data defining a vector orthogonal to the plane, although the process means may, of course, use any suitable representation or encoding.

The system may exploit any appropriate definition of what it means for a three- dimensional movement to occur principally parallel to a plane over a time window. In a preferred set of embodiments, the system uses principal component analysis to determine the plane. A succession of position estimates for the input object over the first time period may be determined (e.g. at a number of evenly-spaced intervals), and suitably analysed, e.g. using singular value decomposition. A set of n three-dimensional coordinates representing a succession of n position estimates for the input object may be determined by the system. These

coordinates may represent the centroid of the input object, or one or more points on the surface of the object, or may represent its position more generally or vaguely. The coordinates may only be approximations to the true positions.

The coordinates can conveniently be represented in an 3 x n matrix, P.

A singular value decomposition is a matrix factorisation which factorises the matrix P into three matrices U, S and V, such that P = USV^T, where the matrix S is an 3 x n diagonal matrix containing the singular values in descending order along its diagonal. The 3 columns of the 3 x 3 matrix U are called the left singular vectors while the n columns of the n x n matrix V are the right singular vectors. The first column of the matrix U will be a three-dimensional vector indicating the direction which is most significant in the set of coordinates (i.e. the direction of maximum variance); its second column will contain the direction which is the next most significant, while the third column gives the least significant direction. The column vectors of U are orthogonal, so the vector in the third column is normal to the plane containing the first and second column vectors.

In detail, the system preferably centres the position data by subtracting the mean coordinate of the set from each of the coordinates in the set. It preferably performs a singular value decomposition on the centred data. It may check whether the movement is substantially planar by determining whether the largest singular value and the middle singular value are both significantly higher than the smallest singular value according to respective predetermined significance criteria. The criteria may, for example, indicate significance if the largest singular is at least four times greater than the smallest value and the middle value is at least twice the smallest value. Of course, alternative criteria may be used.

If the movement is determined to be substantially planar, the system may determine a vector which is normal to the principal plane of movement, and this vector may be the characteristic (or a component thereof) which is then used to determine the second input. This vector may be the third column of a matrix defined as the matrix U above.

It will be appreciated that coordinates, vectors and matrices are mathematical or logical constructions, and no particular limitation is to be inferred as to how they might be represented and manipulated by the processing means or in any associated memory.

To give a practical example of an interaction with a system embodying the invention, the input object may be a user's hand, which, during the first time period, is moved in a circular motion, substantially parallel to a display screen (e.g. an LCD panel on apparatus embodying the system).

The first two columns ui , u₂ in the matrix U, as defined above, are then vectors lying in the x - y plane, substantially parallel to the screen, whereas the last column u₃ will lie in a z-direction substantially perpendicular to the screen.

The system may determine a plane which is normal to the z-axis as the plane of interaction. This may be any normal plane, but is preferably determined to be the plane which contains the mean coordinate of the set of position estimates.

Once a plane is determined, the system may use this information in any appropriate way for determining the second input to the system. Various possibilities are described in more detail below.

In one set of embodiments (not necessarily limited to those in which the

characteristic comprises a plane), the system can limit the set or range of potential inputs to which it may respond during the second time period. For example, a set of potential inputs may be restricted to include only inputs associated with substantially planar movements, or to include only inputs associated with the identified plane, or to ignore components of the movement of the input object during the second time period which are orthogonal to the identified plane. The system can thereby improve robustness by reducing the likelihood of an incorrect detection if the user inadvertently starts to stray out of the identified plane in a way which might otherwise be recognised by the system as a quite different input. ln another set of embodiments, the characteristic represents or comprises a direction along which, or parallel to which, the movement principally or substantially occurs. This characteristic may be particularly useful when the input object can validly be moved substantially linearly (in one dimensional), such as might be the case when interacting with a virtual linear slider. Substantially linear movement during the first time period may be identified by determining that the first singular value of a singular value decomposition of position data, as described above, is significantly larger than both the second and third singular values, according to a predetermined significance criterion (e.g. being at least four times greater than each). The characteristic direction may then be derived from the first left singular vector.

In another set of embodiments, the characteristic comprises the location of a reference point for a path travelled by the input object during (at least) the first time window. It may, for instance, comprise the coordinates of an origin of a circular arc, or the coordinates of a focus of an ellipse.

The system may thus be arranged to determine that the movement of the input object during the first time period matches a predetermined class of path, at least to a substantial or predetermined extent (e.g. within a predefined threshold of deviation, or within a predetermined criterion of statistical significance).

In some embodiments, the class of path may form a component of the determined characteristic of the movement of the input object; i.e. the identification of the class may be used when determining the input to the system.

A class of path may, for example, be the class of all circular arcs having a central angle of at least 90 degrees. Members of the class may have an associated type of reference point or points, such as the origin for a circular arc. The system may be configured to estimate the location(s) of such point(s) from the first signal (e.g. its coordinates in three-dimensional space), and use this characteristic when determining the input to the system. ln some embodiments, the origin of a circular arc gesture may be determined by saving a set of n position estimates for the input object to memory and estimating the centre of these points by taking the mean or median of their coordinates. The system may verify that the position history represents a circular motion by checking that the distance from the estimated origin to each position estimate in the memory is the same, at least to within a threshold tolerance. If this is verified, then the system may save or update the estimated location of the origin for use in processing the second (and possibly subsequent) received signal. The estimated position of the origin may be updated at intervals after the first time period, using subsequently received signals. This updating may be relatively frequent (for example, as often as each time a new position estimate becomes available, which could be around 40 to 120 per second). Rapid updating may be particularly advantageous if the inputs being provided by the user are continuous or real-time inputs, such as rotating a scroll wheel.

In one set of embodiments, the system may be configured to determine that the movement of the object during the first time period has followed a circular arc in a plane (to within a predetermined tolerance) and respond by estimating the origin of the arc. The system may then use a polar coordinate system, centred on the estimated origin, when interpreting movement of the input object during the second time period. For example, the system may disregard radial movement of the input object during the second time period (optionally up to a threshold level, after which the system determines that the continual input has ended), and respond instead only to angular displacement of the input object, e.g. by animating a graphical element on a display screen in angular synchronicity with the input object, and perhaps by increasing or decreasing the volume of an audio amplifier accordingly. In this way, the system can be robust if the user deviates away from a perfect circle, despite his intention to provide a circular-type gesture input to the system.

In another set of embodiments, the characteristic relates to, or comprises, the speed of movement of the input object during the first time period.

Such a characteristic may, in some embodiments, be determined by estimating a start time and an end time of a movement of the input object which corresponds to a predetermined class of movement (e.g. a 'Z'-shaped gesture). The system may then estimate the time it took for the user to perform the gesture from the difference between these times. In some embodiments, this time may be compared with a reference time for the class of movement, to determine whether the movement was faster or slower than typical, and potentially to what degree. The absolute time, or information derived therefrom, may be used when processing the second received signal to determine the input to the system.

If it is determined that the input object moves relatively fast, say, during the first time period, this information may be used to adjust how the processing means responds to movement during the second time period; for example, by increasing the system's responsiveness to faster movements and decreasing its sensitivity to slower movements. In another set of embodiments, the characteristic relates to, or comprises, the scale or extent of the movement of the input object during the first time period.

This may be determined by computing a minimum bounding box, cube or sphere for the movement of the input object over the first time period. Alternatively, where the system is arranged to identify the movement as belonging to a predetermined class of movement or gesture, it may also be configured to store information relating to the movement of the input object during the first time period, such as a series of position estimate coordinates at intervals. Once it has made a gesture identification (possibly using additional signals relating to movement of the input object occurring after the end of the first time period), the system may review the stored information in order to determine a scale for the gesture. This determination may make use of a predetermined parameterisation or scale for the particular class of gesture. In some embodiments, the system may determine an orientation in space for the gesture (e.g. suitable x, y and z axes) and then take the difference between the smallest and greatest x coordinate during the time period to estimate a scale along the x axis, and similarly for the y and z axes. Thus a scale or extent of the last performed gesture can be determined.

It will be appreciated that multiple characteristics may be determined using the first signal, which may be used when determining the input to the system. Moreover, a single characteristic (and an associated value thereof) may be multi-dimensional. It may, for example, represent both a principal plane for the movement and the extent or speed of the motion. The characteristic determined from the first signal (i.e. for the first time period) may also be used by the system when determining further inputs to the system, e.g. when processing subsequent signals comprising further reflections from the input object received during subsequent time periods after the first and second time periods. The system may comprise memory (e.g. RAM) in which a history of values for one or more characteristics is maintained.

The system may be configured to communicate to a user (e.g. a person moving the input object) information relating to the determined characteristic of the movement of the input object. This communication may be by displaying content on a display screen. For example, if the user is moving the input object in a circling gesture, a circular graphic (e.g. a wheel) or animation (e.g. a rotating spot) may be displayed on the display screen. The size of displayed content may be determined by the movement characteristic; e.g., the size may depend on a distance between the display screen and a plane in which the circling input object is principally moving (e.g. the mean perpendicular distance from the display screen to the plane).

Alternatively or additionally, the position of displayed content on a display screen may be determined by the movement characteristic; e.g. the origin of the circular graphic or animation may move in response to a change in the position of an origin of the circling gesture.

In some embodiments, the system may use the second signal to determine a characteristic of the movement of the input object during the second time period (and optionally longer). This may be used in conjunction with the characteristic determined using the first signal when determining the input to the system, or it may be saved and used for processing one or more subsequently received signals.

This characteristic determined using the second signal may be a different type of characteristic from that determined from the first signal, but is preferably the same type; it may constitute an update to the value of the characteristic determined from the first signal. The system may maintain and use a history of characteristic values for a plurality of signals, each corresponding to a respective time period, or it may maintain a single value which is updated (overwritten) at intervals when a new value becomes available. The input object may be any suitable object, including a stylus, a fingertip, a finger, a hand, an arm or a human torso or body, or a group of two or more of these.

Preferred embodiments are arranged to respond to movements of a human hand.

The system may be arranged to respond to movement of two or more input objects which may be moving substantially independently of each other. A movement of just one of them may be associated with a particular input, or the synchronised movement of two or more input objects may determine a particular input.

The system may be configured to determine the input using the second received signal (and possibly also the first received signal) by comparing the movement of the input object as represented in the received signal(s) to a set of template movements or reference gestures (e.g. each representing a particular type or class of movement, such as a clockwise, circular rotation). These reference gestures may be parameterised, e.g. by one or more factors such as speed, size, orientation, etc. In other words, the characteristic of the movement of the input object determined from the first signal may be used to determine one or more of the parameters for the parameterised set of template movements or reference gestures. The Applicant has recognised that determining an optimal set of parameters for reliable performance is not always straightforward. The optimal set may be user-dependent and may change from one user session to the next.

Nevertheless, the rate at which the parameters might optimally change in order to maintain optimal performance is typically slower than the user movements themselves. The parameters can therefore advantageously be estimated and updated over the course of one or several user input movements in order to assist with reliable gesture recognition.

By way of example, if a user starts performing an input gesture principally in a vertical plane, but then, over the course of time, inadvertently allows the plane of the movement to tilt progressively forwards when providing further inputs (or while continuing a continuous input), a system with fixed parameterisation of its gesture sets might reach a point where it can longer accurately identify the user's intended inputs, because the quality of match between the user's movement and a candidate gesture class in the vertical plane may diminish to such an extent that a threshold match grade criterion is no longer met. However, embodiments of the present invention can potentially recognise that the principal plane of the user's movement is shifting over time, and adjust the parameters for the relevant gesture sets accordingly, so that the system responds to a gesture in the angled plane in the same way as it would if the gesture were being performed in the vertical plane under the default parameters.

In some embodiments, a point may even be reached where, if the user were abruptly to revert to moving the input object in the vertical plane ,the system would no longer recognise such movement as the intended input. Other embodiments may adjust parameterisations so as to broaden certain classes of gesture in response to one or more determined motion characteristics, rather than redefining a gesture class in such a way that it might exclude at least some former members of the class.

In general, the system may be such that, for some movements of the input object, the characteristic determined using the first signal influences what input is determined for a given second received signal. This would preferably be the case even if, as in some embodiments, the first signal were not used directly when determining the aforesaid input to the system from the second received signal. In other words, a different input might be determined depending on the movement of the object during the first time period. One of these different inputs might be a null input; i.e. the system not recognising a valid input at all. The system may thus recognise a particular movement of the input object during the second time period as a legitimate input, when this movement would otherwise have been ignored, had the appropriate characteristic not been determined from the first signal.

The transmitted and/or received signals may be electromagnetic signals, e.g. light or radio waves, such as for use in optical or RADAR positioning. The system may comprise one or more optical cameras, for example. However, the signals are preferably acoustic. Acoustic signals travel much more slowly than electromagnetic signals, and are thus better suited to being processed to determine motion information on resource-constrained devices such as mobile telephones.

Particularly preferably, the signals are ultrasonic since these are undetectable to the human user; i.e. having a frequency higher than the normal human hearing range; typically this means the signals have a frequency, or base or median frequency greater than 20 kHz, e.g. between 30 and 50 kHz or higher.

The system may be distributed (e.g. with the processing means being remote from the receiving means), but is preferably implemented on a single apparatus or electronic device, such as a personal computer, laptop computer, mobile telephone, car dashboard, washing machine, etc. Embodiments of the invention can provide a touchless user interface for an electronic device; i.e. a user interface in which one or more features of the device can be controlled through appropriate movements of an input object; typically part of a user's body such as a hand. The signals may originate from an external source, but are preferably emitted from the apparatus itself. The apparatus may therefore comprise one or more signal transmitters, such as an ultrasonic transmitter. It preferably comprises one or more signal receivers, such as ultrasonic microphones, which may be separate from any transmitters or may share some or all components in common therewith.

Any transmitted signal may be transmitted continuously or discretely. A signal may, for example, comprise a series of discrete, acoustic chirps, i.e. rising or falling tones. Reflections from different parts of the input object will be received at different times and may exhibit different characteristics; such differences may be used to distinguish the input object from other objects from which reflections of the signals are received. Techniques for motion-based input control, which may be used in embodiments of the present invention, are described in the applicant's earlier applications WO 2006/067436, WO 2009/122193 and WO 2009/1 15799, each of which is hereby incorporated by reference in its entirety.

The processing means may comprise any combination of central processing units, microprocessors, graphical processing units, DSPs, FPGAs and/or ASICs. It may also comprise storage means or memory, such as RAM, ROM, flash, EEPROM, magnetic media, etc. It may be in an single apparatus (e.g. inside an enclosure or casing) or distributed. The system may comprise a network. The nature of the receiving means will depend on the type of signal, but may include an electromagnetic or acoustic receiver; e.g. comprising one or more cameras, photodiodes, radio receivers, RADAR receivers, LIDAR receivers, microphones, microphone phased array.

The system may respond in any appropriate way to the input: it may write data to memory or transmit data (e.g. over a data network); it execute a software function; it may activate or update a visual interface such as a light or a display panel; it may emit an acoustic signal; it may move an actuator; etc.

In some embodiments, the system may be able to determine a characteristic of the movement of the input object while in a relatively low-power-consumption state (e.g. in sleep or standby), and/or while in a relatively unresponsive state (e.g. in a "locked" state, in which the system is receptive only to a limited set of inputs). It may be configured to enter a relatively high-power-consumption state and/or a relatively responsive state, in response to the first signal. These state changes may affect hardware and/or software; for example, a CPU may enter a higher power state, or a display screen backlight may turn on, or a software operating system may enter a different state.

Preferably, the system is configured to enter the relatively high-power-consumption and/or responsive state before determining the input to the system using the second received signal. For example, the system may determine motion of the input object within the first time window to be occurring within a predetermined distance of the system, or in a particular direction, and this may trigger the system to ready itself for more complex user interactions. In this way, the responsiveness of a system can be increased when waking from standby (or from a locked state) in response to a movement of an input object, because a characteristic of the movement of the input object from the "wake" phase can also be used to determine a subsequent user input, such as loading a particular software application.

For example, if the system is embodied in a mobile telephone, when the mobile telephone is in a conventional "locked screen" state (to save power and avoid accidental dialling), it may be configured to be woken by a user gesture performed near a display screen. If it is woken by a circular gesture, a plane in which the gesture is principally carried out may be determined while the telephone is leaving the "locked screen" state. Knowledge of this plane can then be used to interpret subsequent gesture inputs. This can reduce the time taken before a useful input action can occur.

Optional or preferred features of one aspect or embodiment described herein may, wherever appropriate, be applied to any other aspect or embodiment. Where reference is made to different sets of embodiments, it should be understood that these sets are not necessarily distinct but may overlap.

Certain preferred embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

Figure 1 is a figurative front projection of a display device embodying the invention, showing a user interacting with the device; and

Figure 2 is a representation of a user interacting with the device while an interface guide is displayed on the display device.

Figure 1 shows a display device having a flat-panel display screen 2 mounted vertically in a rectangular frame 4. Mounted centrally in the top beam of the frame 4 is an ultrasound transmitter 6. Mounted in the frame 4 are a top-left ultrasound receiver 8, a top-right ultrasound receiver 9, a bottom-left ultrasound receiver 10 and a bottom-right ultrasound receiver (not visible). Although only four receivers, or microphones, are shown, there may of course be more transmitters and/or receivers.

The display device contains memory, a processor running software, display drivers, etc. familiar to one skilled in the art. A user's hand 12 is shown in a succession of time-separated positions 12a, 12b, 12c as it moves from right to left past the display 2, being first in front of the display 2 to the right, then central, and finally to the left.

The transmitter 6 emits a series of ultrasonic signals or a continuous, varying signal, which are reflected off the hand 12 when it is at the various positions 12a, 12b, 12c. The reflections are received and sampled using the ultrasound receivers 8, 9, 10, each of which defines a separate channel.

Although only three positions for the hand 12 are shown, position estimates are normally obtained at a higher rate, e.g. on the order of one every few milliseconds. Rather, there may typically be hundreds or thousands of samples taken during this time.

The device processes the signals received at the receivers 8, 9, 10 to determine a succession of position estimates for the hand 12 as it moves past the display 2. These may be determined by determining a round-trip time of flight for the transmitted signal for each channel, and then estimating the intersection point of three or more ellipsoid surfaces defined by the respective times of flight for the respective channels and the positions of the transducers in the frame. Such calculations will be familiar to one skilled in the art.

Tracking the user's hand 12 for some period of time results in a set of N recorded positions p,- }^ . These may be determined at regular intervals; e.g. 100 Hz.

Suppose that these positions lie in a two dimensional plane. In this case there exists a coordinate system C such that the positions can be specified by only two coordinates. The origin o of this coordinate system can be taken as the mean of the recorded positions, i.e. .

To find the orientation of the coordinate axes the singular value decomposition (SVD) is instrumental. The SVD of a matrix A e m^{K xL} is of the form A = U∑V^T where U e m^KxK and V e m^LxL are orthogonal and∑ e m^KxL is diagonal (i.e.

∑,. _j = 0 whenever i≠ j ). The columns of U and V are the left and right singular vectors of A respectively and the diagonal elements of ∑ are the singular values. Without loss of generality one can assume that∑_{l l}≥∑_{2 2} > ... > 0 .

The coordinate axes are identified through the following operations:

1. centre the position data: p\ = p_t - o , = 1,2, ... , N ; 2. construct the data matrix P = (p p₂' · · · p_N' ) ;

3. compute the SVD of P , i.e. P = U∑V^T ; and

4. set u_i equal to the z^' th column of U . The two first left singular vectors, u_l and u₂ , will be parallel to the plane containing the recorded position data, while the third vector will be orthogonal to this plane. Thus, each position point can be written in the form p_i = o + a_iiu_l + _i2u₂ , z^' = l,2,...,N where oc ,a_i2 are constants specific to each point and u_l, u₂ are the first singular vectors of P .

In practice the user movement cannot be expected to lie perfectly in a plane.

Additionally, the position estimates may not be wholly accurate. However, the plane P obtained above can be said to be the best fit to the obtained data.

Specifically, if d(p_t,P) denotes the minimum distance between p_i and P then the quantity

is minimised over all possible planes containing the origin o . The error ε equals the square of the smallest singular value of P . Therefore if ∑_{3 3} is significantly smaller than∑_n and∑_{2 2} , the user movement lies in the plane to a significant degree. This significance threshold may, for example, be met if ∑_{3 3} is less than a quarter of both∑_n and∑_{2 2} .

In some cases it is preferred to update the orientation of the plane continually as new position data become available. To this end it can be beneficial to compute the SVD of the matrix P = PP^T instead of P . This give identical results since the left singular vectors are the same for both matrices, but only requires the calculation of the SVD of a 3x3 matrix as opposed to a 3xN matrix. A recursive update of P can be implemented as follows. 1. Initialization step (k = N): 1 ^N 2. Update step (k > N):

G = (\ - )G_k__{1 + Pk}p_k ^T ; and β is a constant selected appropriately from the range [0,1] , where a larger value of β gives a faster update (i.e. greater weight to more-recently-received position data).

Once determined, the plane of interaction can be used to interpret subsequent movements of the input object. For example, future position data may be projected onto the plane and compared against a library of candidate planar gestures, to determine which candidate gesture most closely matches the data. If the user's hand 12 is found to have moved dramatically away from the plane (i.e. by more than a threshold amount), the system may cease projecting the data onto the plane and seek instead to match the movement to a (larger) library of three-dimensional gestures. However, while the planar projection is maintained, the system will be able to provide more reliable matching because it is searching a smaller set of candidate gestures (i.e. planar gestures). The set of candidate gestures and associated inputs may be further limited by the initial orientation of the interaction plane. For example, a set of gesture inputs in the vertical plane may differ from a set of gesture inputs in the horizontal plane. If the initial movement is determined to be substantially vertical, the system may limit its matching to the associated set of gesture inputs, and may continue to limit to this set even as the plane of interaction alters somewhat over time. In an extreme example, it could be that the vertical gesture set might end up being applied to movements in a 45 degree inclined plane, while, if the initial interaction had been horizontal and the interaction plane had slowly shifted to a similar 45 degree angle, the (different) horizontal gesture set might have been applied, such that the very same movement could have caused a quite different response in the device. The device may similarly limit or adjust the parameters of one or more candidate gestures (each representing a class of gesture) based on a relative speed of movement of the hand 12 determined from the historic position data p,- }^ . A scale or extent of movement may also result in a similar limiting or

reparameterisation.

Figure 2 shows graphical icons being displayed by the display screen 2 which invite the user to interact with the device in one of six different ways: by performing a continuous forwards or backwards circling motion about one of an x, y or z axis, where the x and y axes are parallel to the surface of the display screen 2 and the z axis is orthogonal to it. These gestures can represent six classes of gesture input, each of which may be associated with a different response by the device; e.g.

advancing incrementally forwards or backwards through a list of photographs to be displayed on the display screen 2; gradually increasing or decreasing screen brightness; and gradually increasing or decreasing screen contrast.

In a naive implementation, not embodying the present invention, the device might continually analyse the set of the last N position estimates p,- }^ for the user's hand 12 to determine a closest match at any given time to one of the six gesture classes. The system might activate and animate the relevant control icon, and also perform the associated functional response. However, if the user's movement deviates away from one of the Cartesian planes, and/or away from following a perfect circular arc, a different one of the control icons may abruptly become active and the device may perform a quite different action from that intended by the user (e.g. increasing the display brightness when the user desires to scroll backwards through a digital photo album).

Instead, an embodiment of the present invention can provide inertia against such changes by adapting (continually or discretely) to a characteristic of the user's input, such as his speed and angle of interaction. If the user's hand 12 drifts gradually away from circling about the x axis, for example, to circling about a vector at an angle to the x axis, the system can detect this and reparameterise the gesture classes (or equivalently, modify the position data) so that the currently vector becomes the x axis and also redefines the y and z axes for the purposes of matching the movement to one of the six gesture classes.

Thus, an input system has been described and exemplified which can reduce the incidence of missed or misclassified inputs, thereby increasing detection accuracy and user acceptance.

References herein to prior art do not constitute an admission that such publications form part of the common general knowledge in the art in any country of the world. The word "comprise", and variants thereof such as "comprises" and "comprising", as used in the present description and any accompanying claims, are used in an inclusive or open sense (i.e. so as not to preclude the presence or addition of further features), except where the context requires otherwise due to explicit language or necessary implication.

Claims

1. A method of processing a user input to a system, comprising:

receiving, during a second time period, starting after the first time period has ended, a second signal comprising a further reflection from the input object;

using (i) the second received signal, and (ii) said characteristic, to determine an input to the system; and

responding to the input.

2. A method as claimed in claim 1 , comprising determining the characteristic of the movement of the input object from data representing the position of the input object at two or more moments during the first time period.

3. A method as claimed in claim 1 or 2, wherein determining the characteristic of the movement of the input object comprises applying one or more filters to a representation of the first signal.

4. A method as claimed in any preceding claim, wherein the characteristic of the movement comprises one or more of: (i) the speed or velocity of the object in three dimensional space; (ii) a component of the speed or velocity of the object in a particular direction, plane or surface; (iii) the direction of movement of the object; (iv) a plane within which, or parallel to which, the movement principally or substantially occurs; and (v) the extent or scale of the object's movement over a time window

5. A method as claimed in any preceding claim, wherein the second signal is not used for determining the characteristic.

6. A method as claimed in any preceding claim, wherein determining the characteristic of the movement of the input object comprises using principal component analysis to determine a plane, parallel to which the movement of the input object during the first time window principally occurs.

7. A method as claimed in any preceding claim, wherein determining the characteristic of the movement of the input object comprises determining a set of three-dimensional coordinates representing a succession of position estimates for the input object.

8. A method as claimed in claim 7, comprising applying a singular value decomposition to the set of three-dimensional coordinates, or to data derived therefrom.

9. A method as claimed in claim 8, comprising determining whether movement of the input object is substantially planar, by determining whether a largest singular value from the singular value decomposition and a middle singular value from the singular value decomposition are both significantly larger than a smallest singular value from the singular value decomposition, according to respective predetermined significance criteria.

10. A method as claimed in any of claims 6 to 9, comprising using said plane to limit a set or range of potential inputs to which the system may respond.

1 1. A method as claimed in any preceding claim, wherein the characteristic of the movement of the input object comprises a direction along which, or parallel to which, the movement principally or substantially occurs during the first time window.

12. A method as claimed in any preceding claim, wherein the characteristic of the movement of the input object comprises the location of a reference point for a path travelled by the input object during the first time window.

13. A method as claimed in claim 12, comprising estimating the origin of a circular movement of the input object, by saving a set of position estimates for the input object during the first time window to a memory, and calculating a mean or median of the position estimates.

14. A method as claimed in claim 13, comprising verifying that the saved position estimates represent a circular motion by checking whether the distance from the estimated origin to each position estimate in the memory is the same, at least to within a threshold tolerance.

15. A method as claimed in claim 13 or 14, comprising using subsequently received signals to update the estimated position of the origin, at intervals after the first time period.

16. A method as claimed in any preceding claim, comprising determining that the movement of the input object during the first time period matches a

predetermined class of paths, at least to a predetermined extent.

17. A method as claimed in claim 16, wherein the characteristic of the movement of the input object comprises said determined class of paths, and wherein the method further comprises using the identity of the class when determining the input to the system.

18. A method as claimed in any preceding claim, wherein the characteristic of the movement of the input object relates to speed of movement of the input object during the first time period.

19. A method as claimed in claim 18, wherein a movement of the input object during the first time period corresponds to a predetermined class of movement, and wherein the method further comprises estimating a start time and an end time of this movement.

20. A method as claimed in any preceding claim, wherein the characteristic of the movement of the input object relates to the scale or extent of the movement of the input object during the first time period.

21. A method as claimed in any preceding claim, comprising communicating, to a user, information relating to the determined characteristic of the movement of the input object.

22. A method as claimed in any preceding claim, comprising using the second signal to determine a characteristic of the movement of the input object during the second time period.

23. A method as claimed in claim 22, comprising using the characteristic of the movement of the input object during the second time period to update a value of the characteristic determined from the first signal.

24. A method as claimed in any preceding claim, comprising estimating a trajectory of the input object from the second signal; using the characteristic of the movement of the input object during the first time period to modify the estimated trajectory; and using the modified trajectory when determining the input.

25. A method as claimed in any preceding claim, wherein determining an input to the system comprises comparing the movement of the input object as

represented in the second received signal to a set of template movements or reference gestures.

26. A method as claimed in claim 25, wherein the set of template movements or reference gestures is parameterised, and wherein the method further comprises using one or more user input movements to estimate and update parameters for the set of template movements or reference gestures.

27. A method as claimed in claim 25 or 26, wherein the set of template movements or reference gestures is parameterised, and wherein the method further comprises using the characteristic of the movement of the input object during the first time period to set one or more of the parameters for the template movements or reference gestures.

28. A method as claimed in any preceding claim, wherein the first and second signals are both acoustic signals

29. A method as claimed in any preceding claim, further comprising processing the first signal to determine a further input to the system.

30. A method as claimed in any preceding claim, comprising determining the characteristic of the movement of the input object while the system is in a relatively low power-consumption state or a relatively unresponsive state, and causing the system to enter a relatively high-power-consumption state or relatively responsive state in response to receiving the first signal.

31. A system for processing a user input, comprising receiving means configured to receive signals through air, and processing means configured to: process a first signal, received during a first time period, comprising a reflection of a transmitted signal through air from a moving input object, so as to determine a characteristic of the movement of the input object;

32. A system as claimed in claim 31 , further configured to carry out a method as claimed in any of claims 1 to 30.

33. A software product comprising instructions which, when executed by processing means, cause the processing means to:

respond to the input.

34. A software product comprising instructions which, when executed by processing means, cause the processing means to carry out a method as claimed in any one of claims 1 to 30.