WO2005081096A2

WO2005081096A2 - Control system for computer control devices

Info

Publication number: WO2005081096A2
Application number: PCT/GB2005/000537
Authority: WO
Inventors: Riccardo Poli
Original assignee: Wivenhoe Technology Limited
Priority date: 2004-02-17
Filing date: 2005-02-15
Publication date: 2005-09-01
Also published as: WO2005081096A3; GB0403416D0

Abstract

A computer control system to predictively carry out instructions selected through an on-screen environment includes a selection means to select on-screen elements corresponding to the instructions. Data is recorded on the location of the selection means and on the selection of an element, which data is analysed and used to determine the likelihood that the user intends to select an instruction. The system is subsequently capable of implementing said instructions if preset criteria are fulfilled. To reduce the risk of an error being made and of incorrect instructions being carried out, the data is statistically analysed, with no instruction being implemented by the system before a pre-set level if data is availabe. The system is particularly suitable for use with a standard mouse-based selection means.

Description

CONTROL SYSTEM FOR COMPUTER POINTING DEVICES

Field of the Invention

The present invention relates to a device for indicating and selecting an on-screen function to control the tasks performed by, for example, a computer. The invention further relates to a method to enable said indication and selection.

Background to the Invention

From their early development, certainly since the late 1960s and early 1970s, computers have been accessible to the user, enabling interaction and selection of functions, through the use of a full-screen editor. The majority of full-screen editors, function by the user moving a pointer, physical or virtual, to a defined area of the screen and then clicking or otherwise selecting that area causing the computer to perform a predefined task. Often the area selected is covered by an on-screen icon to indicate the function attributed to that particular area of the screen.

A number of devices have been developed to function as pointer and enable the user to indicate a particular area. Such devices include light pens, use of a keyboard to move a cursor on the screen, touch screens and graphics tablets. The most popular device has proved to be what is commonly known as a mouse. Movement of the mouse across a surface causes an on-screen indicator to move. An area on the screen is selected by clicking one of a number of buttons, normally located on the upper surface on the mouse.

The use of a mouse has however a number of limitations. First, frequent use can give rise to repetitive strain injury. Secondly, speed and fine control can often be lacking. Thirdly, mice are difficult to use for those with hand or other upper limb disabilities.

It is therefore desirable for the system for the selection of on-screen elements to include a predictive element to determine when a user wishes to select a particular function. It is further desirable to provide a selection system such as a mouse or other such device which can provide selection of on-screen functions more rapidly and accurately than is presently the case.

Summary of the Invention

According to the invention there is provided a selection system to predictively carry out a selection; the system comprising:

a selection means for selecting an action to be performed;

on-screen elements corresponding to an action to be performed, selection of an element by the selection means causing the action to be performed;

location-recording means to record data concerning the location of the selection means;

selection-recording means to record data concerning selection of an element;

analysis means to analyse the data and to determine the likelihood that a user intends to make an action;

implementation means to effect an action predicted by the analysis means, without the user selecting an action.

The system provides for a predictive system, based on established results, to predict and carry out an action it is calculated a user intends to carry out. Selection of actions is therefore speeded up and the risk of errors is minimised. Preferably the data from the selection and location recording means are stored in an appropriate database. The data can therefore be easily recalled and manipulated.

The implementation means is preferably enabled when a minimum pre-set amount of data has been stored. The risk of an incorrect selection being predicted and brought into effect is thereby minimised.

The analysis means conveniently carries out statistical analysis on the data to determine the user's intention.

Preferably, the selection means is a mouse, having one or more buttons by which a user makes a selection. Particularly preferably, the mouse governs motion of an on-screen pointer selecting an on-screen element, often in the form of an icon.

Advantageously, the trajectory of the on-screen pointer is stored. Particularly advantageously, data on the frequency of motion of the on-screen pointer from one element to another is stored.

The system advantageously includes a database of images around a portion of a screen selected by a user. Particularly advantageously, images of the screen neighbouring the image around the pointer are also stored in a database.

Preferably the system includes a comparator to compare the image around the area selected by the pointer with those images in the image database. Conveniently comparison is carried out by a comparison of the pixels making up the images, a cumulative difference between the pixels being determined, whereupon those images having a higher cumulative difference than a preset value are stored. Selection data is preferably associated with each image storing the number of times an image is selected and also the number of times an image is not selected.

Detailed Description of the Invention

In the following description a predictive selection system is used in a windows based environment, described with reference to a mouse-driven on-screen pointer.

It will be understood however, that the system can be implemented in connection with other selection means and non-windows based systems.

The system is designed for use in conjunction with a screen through which a user can select commands instructing a computer to carry out particular tasks. In a windows environment the screen is divided into a number of virtual areas, each area corresponding either to a particular command or to no command. The user positions a virtual pointer over the area which corresponds to the task he wishes to perform and selects that area. In order to assist identification of the correspondence between areas and commands, appropriate text or a symbol - referred to normally as an icon - specific to the command is positioned over the relevant area.

The most widely known means by which the virtual pointer is moved across the screen is by use of what is commonly referred to as a mouse. A typical mouse determines its own motion across a surface and converts this to motion of the virtual pointer, by means of specific subroutines in the computer. To select a command the user simply moves the mouse causing the virtual pointer to move across the screen until the pointer is positioned over the on-screen area corresponding to the command. Selection of the command on the screen is through operation of one or more buttons often found on the upper surface of the mouse.

The system described herein is based on the above, but provides, additionally, means by which the actions of the mouse can be made predictive, in the sense of anticipating what the user intends to do and carrying out the intent without the user having to complete the movements required when using a normal mouse. In order to enable the system to achieve this, a learning program is included on the computer which effectively enables the mouse to learn.

The functioning of the learning program can be split into three modules which interact with each other to give the required performance. The three modules are conveniently designated the main module, the learning module and the action module.

The main module is similar to modules incorporated to drive a standard mouse, and comprises an initialisation sub-module to initialise the local and global variables by the use of default and command-line parameters. The sub-module also initialises the graphics system, creates a program window and sets up connection with the mouse itself.

The main module also includes a further sub-module which is responsible for a number of sub-functions as follows. Firstly a sub-function which accepts corrections from the user when the mouse pointer is in a particular program window. Corrections can be initiated by clicks performed by the user on the mouse or specific keys pressed while the window is active. A second sub-function redraws the afore-mentioned window when the window is re-exposed. This can occur after the window has been covered over by another window, or has been moved or minimised.

A third sub-function processes data packets received from the mouse. The data packets enable the velocity vector and the state of the buttons on the mouse to be determined. The motion and state of the mouse is then further processed to produce a corresponding movement of the on-screen pointer on the screen. In addition to the above, the system of the present invention provides that where a button on the mouse has been pressed, and the pointer is not in the program's window, then the learning module is invoked and a signal is sent to the Operating System (OS) that a button has been pressed.

A further sub-function invokes the action module (described later in more detail) if, within a preset time period and following motion of the mouse, no further actions have taken place.

In order to be able to predict what action the user intends to perform next, the system includes a means of accumulating data on how a user uses a mouse, and a means of analysing the data to calculate a series of performance parameters of a user. These parameters are updated within a learning module which performs a number of sub-functions. In order to be able to learn, the learning module maintains a database of already encountered functions and situations (often termed memories) against which the ongoing situation can be compared.

Each memory in the database comprises two elements. The first element includes the images of the on-screen area surrounding the mouse pointer at the time the memory was acquired. The second element is a set of statistical parameters derived from the behaviour of the user. For example, a first parameter records the number of times a user did not click when the mouse pointer stopped in areas of the screen closely resembling the current memory. Further parameters represent the number of times the user did click on each of the available mouse buttons when in said on-screen areas.

When the pointer is in a particular position a comparison is made between the image surrounding it and the database of images described above. Should the image of the present position be sufficiently different, as determined by a computer sub-routine, from those in the database, the new image will be added to the image database. In the alternative, if the image of the present position is not sufficiently different, a new memory will not be created but instead the statistical parameters for that image will be updated.

The third module, the action module, is responsible for automatically simulating the operation of one of the mouse buttons by sending appropriate signals to the

Operating System. Whether or not the operation of a button is simulated depends on a number of factors. Firstly, the system determines if sufficient data has been accumulated to enable an accurate evaluation of the intention of the user to be made. Should this not be the case, then the system will not take an automatic decision. On the other hand, if the system determines there is sufficient data, the action required to bring into effect the predicted intention of the user, will be passed to the Operating System to carry out.

Having described the broad principles of the system, each of the different modules will now be described in more detail. In the more detailed description, the system will be described as being implemented under Linux/Unix in the X- Window System Version 11 (XI 1 hereafter) graphical environment. Implementation is suitable using the C programming language, but the description below is language independent.

In the system a number of variables and constants can be defined which are used by the three modules described above. For example, the width and height of the main window used for displaying program actions, and the database of memorised images can be stored. The memorised images are in effect subsets of the set of image characteristics for the particular window in use.

Further data which are stored in relation to the images are their dimensions, the maximum number of images which can be held in the database, and the minimum number of images required to be in the database, before the system automatically issues commands corresponding to the predicted action of the user.

Data is also stored on how long the delay is from when a mouse stops and the system should issue a click when the system is fully operative. A delay time is also stored for the phase in which the memory database is being constructed, where the delay time is normally longer than the time delay during full operation of the system.

Additional global variables stored relate to the image which is stored. Firstly an image comparator compares the difference between a stored image and the image around the current pointer position. The difference is assigned and compared with a stored threshold value. Should the difference be greater than the stored threshold value the image is added to the stored database of images.

In order to increase the accuracy of the system, neighbouring images, within a pre-set distance from the pointer's co-ordinates are also stored within the database when a new memory is acquired.

The name of a file is stored in which the database of previously acquired memories resides together with its location after execution of the program is completed. The names of the file/device from where the stream of data from the mouse will be read and of the file/device used for output (for debugging purposes) are also defined.

A number of variables are also stored holding XI 1 -related information (such as the display pointer, windows ID's, co-ordinates of the pointer, marks, graphic, contexts etc.). Also pertaining to XI 1 are arrays of XI 1 images relating to the database of images together with corresponding statistics relating to when a mouse button was clicked when the pointer stopped on the images. A record is also maintained on the last button clicked for all images on the database. Allied to this is a flag that indicates whether, the last time the mouse stopped, the system acquired a new image or updated the statistics in the database. A small set of variables are also stored for handling command-line parameters. With regards to the modules, each sub-module has a set of variables associated therewith, which are detailed below.

The variables and constants associated with the main module can be split into two sub-sets. The first sub-set is concerned with initialisation. During the initialisation steps, local variables are defined, where appropriate. These local variables include file descriptors, flags, variables describing the state of the mouse and the position of the pointer on the screen, XI 1 related variables etc. A connection to the XI 1 server is defined and established. Global variables are set to their default values.

In connection with the mouse a simple window which is capable of handling mouse and keyboard events, in addition to redraw requests is established. The connection is opened and set up with the mouse and the current position of the printer on the screen established.

Command-line parameters are processed, which can be used to change program parameters from the acquired default values. The command-line parameters can also be used to load an existing database of memories and associated clicking behaviours from a file. If the "-h" command-line option is used a help screen is provided.

A main-loop sub-routine maintains a second sub-set of variables and constants and whilst the system is in operation acts to maintain and update said variables. The main-loop sub-routine resets the variables used for timing and the file descriptors for the data stream. If at least one memory is already in the database, then a check is carried out to see if a mouse button has been pressed. If this is the case, and the pointer is pointed at an image in the memory database (which, as mentioned above, is shown in the program's window) then this is an indication that the user wishes to perform a correction to the database. The sub-routine then performs a different action depending on what the user does. If the user presses the left mouse button, the statistics for the image are changed such that the next time the pointer stops in an area of the screen matching this image, the system is less likely to simulate a button click of the same type as the last click for that image before the correction. Alternatively if the right button is clicked, together with depression of the Control key then the image together with associated statistics are deleted from the database and the window redrawn.

If the pointer enters the program's window then automatic clicking is disabled.

Conversely, when the pointer leaves the window automatic clicking is enabled. In order to prevent/undo the effects of an unwanted action having been taken by the system, then depression of the Delete or Backspace key; whilst the pointer is in the program window should undo the effects of the action. In the case where the unwanted action did not require a new memory, then the statistics associated with the memory are simply updated. Otherwise, any new memory is deleted and the window redrawn.

If the "q" key is depressed whilst the on-screen pointer is in the program window, then the image database (if an output file name has been provided) is saved and the program exited. If a window is exposed after having been covered or moved then the window is redrawn. The main loop sub-module also includes means to determine if there are data packets coming from the mouse which need to be processed. If, after having made such a determination, the sub-module determines that no such data packets are present, then a pre-set acquisition delay is initiated to wait for any data packets to appear. Should they so appear, then they are decoded and the velocity of the mouse and the state of its buttons determined. The action then taken depends on whether or not the pointer is in the program window. If the pointer is not in the program window, the sub-module uses the information on the button status to determine its next action. If both buttons have been pressed at the same time, a correction is performed in the same way as if "Delete" had been pressed whilst the pointer was in the program's window.

Should both buttons not have been pressed, and the mouse has moved, then the on-screen pointer is moved and this fact flagged for later use. Then, if a button has also been pressed, the learning module is invoked and the Operating System informed that such a button has been pressed. The pointer motion flag is then reset.

If the pointer is in the program window, the action taken will again depend on the mouse status. If the mouse has moved, the on-screen pointer is moved accordingly. If a button has been pressed, the Operating System is informed and the pointer motion flag reset.

Should, following the expiry of the acquisition delay, no data packets appear indicating no mouse activity, then the action module is invoked and the pointer motion flag reset, if it is already set.

The learning module can also be split into two separate sub-modules, a first which acquires and stores memories and a second which carries out a comparison between on-screen images and those already stored in the database of images.

The memory storage sub-module firstly determines the on-screen pointer position. A check is then run on space available in the database and the proximity of the onscreen pointer to the edge of the screen. If both of these parameters are within acceptable values, a tentative memory is acquired. The tentative memory is a combination of the image centred around the on-screen pointer position together with a record of the click type, if any, associated with such an image. A check is then carried out which compares the tentative memory against other memories within the database. If there is sufficient difference, such that the tentative memory constitutes a new image, the image is added to the database and the screen redrawn accordingly. On the other hand if the image is not sufficiently different from an already-stored image in the database, the tentative memory is not added to the database, but the statistical data for the already-stored image is up-dated accordingly.

Comparison of images is carried out by means of an image similarity evaluation (ISE) module. The ISE module compares each image in the database, including shifted versions thereof, with the new image. A similarity value is accorded to each image pair. In order to accord a value, corresponding pixels in the two images are compared by evaluating the sum of the absolute differences between the pixels' red, green, and blue components. The resulting values are calculated for every pixel position and are accumulated to yield the similarity value for the images.

The third module present is the action module. In order for the action module to function a number of statistical parameters are kept for each memory in the database. These parameters relate to the following: the number of times a user did not click when the mouse pointer stopped in areas of the screen closely resembling the corresponding database memory; the number of times the user clicked the left mouse button; the number of times the user clicked the middle mouse button (if present); the number of times the user clicked on the right mouse button.

The parameters are used when the system is required to make a decision as to the push of which mouse button, if any, to automatically simulate. A simple way of reaching the decision is to use the action to which the highest probability, as determined by the statistics held in the memory database. However, to ensure that the parameters are statistically valid (in the sense that they have a sufficiently high likelihood of returning the action which is intended by the user), the database must contain a minimum number of images, acquired during training. Moreover, the image around the on-screen mouse pointer must correspond sufficiently closely to an image in the database. Should these conditions not hold, no decision is made by the system.

Whether the system clicks automatically or not the behaviour of the action module is treated exactly as if produced by the user. This means that any action performed will be passed to the Operating System. In addition the action will be used to update the database. The only difference between the behaviour of the action module and that of the user is that automatic mouse clicks produce an audible sound that provides the user with feedback.

The system can also be used in a similar way to that already described, to predict the target which the user intends to reach. In this mode the system would check the trajectory of the on-screen mouse pointer and note the images along and adjacent to the route. The system would use the database to predict where the pointer is likely to end up, when following that trajectory. Based on the known trajectory and the database, the system can then perform either moving the pointer to a location, or clicking at that location or both.

Furthermore, this facility can be used to correct errors in the trajectory, which errors lead to the pointer not being in precisely the correct position when the mouse ceases to move. The predictive element of the system determines in what position the mouse pointer should be and moves the pointer accordingly.

Hysteresis can then be used to give the pointer a useful "home and stick" behaviour. Using the trajectory correction and hysteresis can enable the mouse to be clicked whilst in motion and can also reduce the need for fine hand/eye coordination as the system will correct small errors made.

The system can also be configured to determine when a pointer or a cursor should be made bigger, to disappear or to be parked. Again, a learning element can be introduced either to enable the user to teach the system or to enable the system to learn in analogous ways to those described above, possibly including corrections from the user. It will of course be understood that the invention is not limited to the specific details described herein, which are given by way of example only, and that various modifications and alterations are possible within the scope of the appended claims.

Claims

1. A computer control system to predictively implement instructions via a screen-based environment, the system comprising: on-screen elements corresponding to an action to be performed, and selection means to select an element and cause the action to be performed; location-recording means to record data concerning the location of the selection means; selection-recording means to record data concerning selection of an element; analysis means to analyse the data and to determine the likelihood that a user intends to make an action; implementation means to effect an action predicted by the analysis means, without the user definitively selecting an action.

2. A system according to Claim 1 wherein the data from the selection and location recording means are stored in an appropriate database.

3. A system according to Claim 1 or Claim 2, wherein the implementation means is enabled when a minimum pre-set amount of data has been stored.

4. A system according to any preceding Claim, wherein analysis means carries out statistical analysis on the data to determine the user's intention.

5. A system according to any preceding Claim, wherein the selection means is an on-screen virtual pointer for selection of an on-screen element said element often in the form of an icon.

6. A system according to Claim 5, wherein movement of the on-screen virtual pointer is governed by a mouse having one or more buttons by which a user makes a selection.

7. A system according to Claim 5, wherein, the trajectory of the on-screen pointer is stored.

8. A system according to Claim 7, wherein data on the frequency of motion of the on-screen pointer from one element to another is stored.

9. A system according to any preceding Claim, including a database of images around a portion of a screen selected by a user.

10. A system according to any one of Claims 5 - 9, wherein images of the screen neighbouring the image around the pointer are also stored in a database.

11. A system according to anyone of Claims 5-10 including a comparator to compare the image around the area selected by the pointer with those images in the image database.

12. A system according to Claim 11, wherein a comparison is carried out by a comparison of the pixels making up the images, a cumulative difference between the pixels being determined, whereupon those images having a higher cumulative difference than a preset value are stored.

13. A system according any preceding Claim wherein selection data is associated with each image storing the number of times an image is selected and also the number of times an image is not selected.

14. A system according to any one of Claims 7-13, wherein a predictive element determines the end point of the trajectory of the virtual pointer by statistical analysis of the stored data.