WO2002008881A2 - Human-computer interface - Google Patents

Human-computer interface Download PDF

Info

Publication number
WO2002008881A2
WO2002008881A2 PCT/GB2001/003153 GB0103153W WO0208881A2 WO 2002008881 A2 WO2002008881 A2 WO 2002008881A2 GB 0103153 W GB0103153 W GB 0103153W WO 0208881 A2 WO0208881 A2 WO 0208881A2
Authority
WO
WIPO (PCT)
Prior art keywords
gesture
software
library
gestures
display
Prior art date
Application number
PCT/GB2001/003153
Other languages
French (fr)
Other versions
WO2002008881A3 (en
Inventor
Christopher Worthington
Ian Duncan Campbell Andrew
Original Assignee
Qinetiq Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to GB0017793.1 priority Critical
Priority to GB0017793A priority patent/GB0017793D0/en
Application filed by Qinetiq Limited filed Critical Qinetiq Limited
Publication of WO2002008881A2 publication Critical patent/WO2002008881A2/en
Publication of WO2002008881A3 publication Critical patent/WO2002008881A3/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/038Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/0482Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance interaction with lists of selectable items, e.g. menus
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object or an image, setting a parameter value or selecting a range
    • G06F3/04845Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object or an image, setting a parameter value or selecting a range for image manipulation, e.g. dragging, rotation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for entering handwritten data, e.g. gestures, text
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16Sound input; Sound output
    • G06F3/167Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/038Indexing scheme relating to G06F3/038
    • G06F2203/0381Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/048Indexing scheme relating to G06F3/048
    • G06F2203/04808Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen

Abstract

A multimedia human-computer interface includes a position sensitive panel and a display wherein gestures drawn onto the panel by a user are interpreted by gesture processing software, and, according to the gesture input and the context in which it is used, the operating mode of software for the manipulation of images is affected. Typically the panel will be coincident with the display. A programmable library of gestures is stored in a compute system, each gesture corresponding to a particular software function. Gestures may be programmed or reprogrammed to control any suitable operation of the software. The invention provides for the switching of operating mode, as well as the control of parameters associated with a given operating mode. The invention also provides for speech recognition techniques to be combined with the gesture recognition to enhance the functionality still further.

Description

Human - Computer Interface.

This invention relates to a multimedia human-computer interface, and also to a method of quickly inputting information into a computer. More specifically, it relates to a means of entering commands and data into a computer system using gestures applied to a panel. The invention also allows the operator to use speech to enter commands into the computer. The invention provides an interface between the operator and software running on the computer system, the software generally being connected with image analysis or manipulation. However, any programs having a requirement for selecting and manipulating data displayed on a screen may benefit from use of the invention. Text selection and manipulation may also be performed.

Background And Prior Art

It is an aim of the designer of human - computer interfaces to make them as easy to use as possible. Differing requirements of different users, and applications of computers in very different environments have made the problem even greater. As well as on the office desktop, computers are being used in such diverse areas as operating theatres, factories, etc and by mobile users, with laptop, palmtop and handheld computers, and by people in the home sending messages whilst sat on the settee watching television. All these people have particular requirements for entering data into the computer. Some of these users may have a particular need to concentrate particularly hard on the computer display for some reason, and will find it an inconvenience to look away from the image presented to find a keyboard or mouse. This invention will be of particular use to these people.

It is the job of the image analyst to gather information from an image, and compile a report on it The analyst uses a computer in this task, viewing a digital representation of the image on the computer display. In carrying out their task there may be a great deal of manipulating of the image to be done. For example, the analyst will need to zoom in on areas to see them in more detail, pan around the image if it is too large to fit on the display device, select localised areas for image processing, such as inverting or filtering, move or copy areas to compare against each other or save as reference images, rotate areas, or perform any other function as may be necessary.

Modern computer systems usually have attached, as a means for inputting information and performing this manipulation, a keyboard and a mouse. These tools are fine for many tasks, and work well, but for some tasks other methods of interacting with the computer may be preferable. When working with images on a computer screen the keyboard can be very awkward. The mouse solves many of the problems associated with image manipulation, but introduces problems of its own. It is only when performing a great number of image manipulation tasks that these problems come to light. For example, when it is required to move an object from point A on the screen to point B using the mouse, there are a great many steps to go through, when the problem is broken down. These are:

Locate mouse on desk

Move arm to locate hand over mouse

Grasp mouse Move mouse to visually locate pointer on screen

By moving mouse, move pointer to locate it on object at A

Locate index finger on mouse button

Press mouse button to select object at A

Hold mouse button down to drag object Move mouse to move pointer, dragging object, near to B (ballistic movement)

Move mouse to move pointer to adjust position (fine) of object on to B

Release mouse button to drop object at B

Many of these tasks are trivial in nature, but when repeated often, they can get tiring, and lead to stress for the operator. Some tasks, such as image analysis or image manipulation may involve a great deal of time spent working in this way, staring at the screen and highlighting areas and moving them etc. One of the key aspects of the job of an image analyst is that, when manipulating the images, it is very inconvenient if they need to look away from the screen for ancillary purposes. This disturbs their concentration, and makes the task more of a chore.

One method of solving this problem is to use speech, as discussed in US patent 5600765. Here, the user can designate features on a display using touch and gesture, and, by speaking commands and indicating on the display, can enlarge, or move etc the item selected. Using speech alone for the inputting of commands can become wearisome and in high noise environments the speech recognition system can become confused by the speech input. This system uses gesture input for selecting and designating areas only, and speech only for the issuing of commands. This can be restrictive.

The current invention solves these problems, and alleviates the stress to the operator caused by the mouse and other operations discussed above. It aims to reduce the number of small interactions that the operator needs to do when manipulating images and compiling reports on the images. It aims to provide a quicker, more intuitive interface that is easier to learn and faster to operate.

Statements Of Invention And Advantages

According to the present invention there is provided a multimedia human computer interface that is useable on a computer that has installed software for enabling an operator the ability to directly manipulate images, comprising at least a position sensitive panel and a display, characterised in that one or more gestures applied anywhere on the panel are interpreted by gesture processing software according to the gesture made and are used to control operating modes of the image manipulation software.

Note that where this specification refers to the manipulation of images, it will be understood that an image can also be an image of textual information. Thus, unless stated otherwise, use of the word "image" covers both pictures and text displayed on a screen. The specification also refers to touch sensitive panels, and position sensitive panels. It will be understood that these terms refer to any panel that is capable of detecting the position of some pointer on, or adjacent, to it, and the terms "touch sensitive" and "position sensitive" should be regarded as equivalent.

The current invention allows the operator to select areas on the screen, and to process those areas with only minimal movement of his gaze away from the area of the screen upon which he is currently working. The invention also provides the operator with a choice of media in which to interface to the computer. Thus the operator is able to concentrate for longer periods of time on the main purpose of the task, and not have to waste time and effort on other tasks such as those mentioned above.

There are two categories of gesture that can be used with this invention. The first is known as a mode change gesture. This is used to switch the operating mode of the software from one to another. For example, if the display software is currently in zoom mode, such that it is being used to control the magnification of a portion of the image, and it was then switched into pan mode, this is called a mode change. A gesture used to do this is a mode change gesture. When in a particular mode, any gestures that are used to operate on a parameter of that mode are called mode operation gestures. This is the second type of gesture. Thus a gesture that, for example, selects the zoom factor, altering the magnification of the image, is a mode operation gesture. «...

A mode change gesture, in the context of this specification, can consist of placing one or more pointers, which can be the operator's fingers, in contact with the panel, and then moving either some, or all of the pointers. The movement can either substantially maintain contact with the panel, or it can be off from the panel. The latter case signifies that whatever action is being performed is now completed. This type of gesture can be thought of as having an equivalent effect to the menu items on more traditional human- computer interfaces, such as the Windows operating system from Microsoft® but which of course are chosen in a completely different manner. A mode operation gesture can differ from a mode change gesture, in that it does not necessarily finish once the pointer or pointers are removed from the touch panel. For example, duririg a move operation, a gesture incorporating multiple pointers that is used to simulate picking up an area of the image, moving it, and putting it down elsewhere on the image can lead to the pointers losing contact with the screen during the move phase of this operation. See below for more information on use of multiple pointers.

Mode change gestures may be pre-set, or they may be changed by the user at will. Also, the gesture used to activate a particular function may be assigned by the user according to their preference. Different gestures can be programmed into the computer system by means of a training function. A library of currently programmed mode change gestures can then be created that holds samples of each gesture Here, a user would make several examples of the gesture currently being programmed. All would vary slightly from each other due to an inability to exactly reproduce a previous tracing on the touch panel. However, the gestures would be substantially alike. These gesture inputs would then be stored as templates. Any gesture then input after this training had been completed would then go to a gesture comparison engine, comprising a pattern matching algorithm. If the gesture were substantially similar, it would be able to recognise it as that trained gesture. The function currently assigned to that gesture could then be activated. Spatial information, i.e. the pattern of the reference gestures is stored in the library. Temporal information, relating to the time taken to draw the gesture can also be stored. This allows the software to differentiate between a mode change gesture input, and a mode operation gesture (in cases where the context of the input required does not define the interpretation of the gesture), so long as the latter does not share the same, or substantially similar temporal parameters. The only limitation on use of a mode change gesture is that it must be able to be drawn on the touch panel without removing the pointer, or all pointers if more than one is used, from the screen. The prior art interfaces have menu options on the screen in set places. Even though the user may be able to alter where these places are, by moving menu bars around etc. the user still has to locate the place where he has put the menu item on the screen each time he wishes to use it. The current invention allows the user to draw mode change gestures anywhere on the touch panel. This provides the benefit that he does not need to look away from the particular area of the image he is currently working on to locate a menu option

There are various methods that can be used to compare a gesture input to the system with those stored in the library. These include basic methods like Fourier transform matching, or correlation algorithms, both of which are established methods of pattern matching, and will be familiar to those skilled in the relevant art. Other techniques can be used that look at various parameters of the gesture. Details of the function of these is beyond the scope of this specification, but for more information on suitable algorithms see Rubine, D, "Specifying gestures by example", Computer Graphics, 329-327 ACM SIGGRAPH, Addison Wesley, July 1991. These methods will look at some or all of the parameters that go to making up a particular gesture, and, comparing these against a reference gesture, will come up with a measure of the similarity of the two gestures. This can be represented as a certainty value of the two gestures being a. match. A threshold can be set for the certainty of the match, in percentage terms, and if two gestures are compared and have a certainty value over this given threshold then they will be regarded being a match.

Before this invention, the operator was able to select areas on the screen using a touch sensitive display, but when he wanted to, say, move a selected area, he had to somehow select the operating mode that allowed items to be moved. This normally involved removing one's gaze from the selected area to look for the button or control that activated the move operating mode, or function, or speaking the command into the computer's microphone. The present invention enables, once the area has been selected, the move function to be chosen by means of the operator making a gesture on the screen that represents selecting the move function. Similarly, if the operator wants to copy the selected area, he will make a gesture that activated the copy function of the software. The number of functions that can be so represented is limited only by the number of unique, easily drawn gestures that the operator can devise. Functions that may advantageously be controlled by the present invention include, but are not limited to,

- Move and copy functions as described above

- Rotate functions, whereby the selected area can be rotated at an angle around a point.

- Pan, whereby when only a small part of an image is visible in some viewing window, that part actually visible may be changed, such that any part of the complete image may be viewed.

- Zoom, whereby some selected area can be expanded in size, such that more detail is visible.

- Select, used for the selection, or confirmation of a highlighted area on the screen.

- Image processing functions, such as filtering, inverting, intensifying etc

- Image annotation functions, such as adding labels to areas, or adding shapes such as rectangles or circles.

Note that the term "operating mode" and the noun "function" and their derivatives are used interchangeably throughout this specification.

Labels can be placed on to areas and defined as markers, that act like a bookmark in a book, such that the areas can be accessed by reference to the "bookmark" label.

Note that the starting point used when drawing a gesture can be used to make that gesture unique. For example, if a gesture consisted of drawing a horizontal line on the panel, then the computer system can detect whether that line is drawn from left to right, or from right to left. Depending on the one detected, one of two separate functions can be chosen. This can be achieved by examination of the temporal characteristics of the gesture. The start point will clearly have a timestamp earlier than that of the end point. This can also be achieved merely by looking at the co-ordinates of the points in the array data structure used to store the gesture information. The first co-ordinate in the array will obviously be the starting point, and the last co-ordinate will be the end point.

As stated above, gestures can be made with more than one pointer at the same time. For example, assuming an area of the image has been selected, and the move command chosen, two pointers can be placed on opposite sides of this area and this area moved by moving the two pointers in a similar manner to picking up a normal object between one's fingers and moving it. This method can also be used for copying, rotating, and other actions that may beneficially employ multiple pointers.

The invention may be made even more flexible and easy to use if a speech recognition system is integrated in with the computer system. Using this, there is the option of inputting commands into the system by means of speech as well as gesture. Another benefit of a speech recognition system is that it may be used for the generation of report information spoken by the operator. Whilst analysing the image the operator can speak his findings into a microphone coupled to the speech recognition system, and the verbal report of the operator can be transcribed into a computer file.

The use of speech recognition brings extra functionality to the system. Some functions have more than one parameter that may be adjusted, and speech can be used to select the parameter. For example, if it is required to filter an area to highlight the edges of shapes on the image, the area could first be selected using a mode operation gesture (assuming the system is in "Select" mode), and the "Filter" option then chosen using a mode change gesture while simultaneously saying the word "Edge" into the microphone. In this way, the computer system knows that the edge detection filter is the one required rather than any other filter that may be applied.

As another example, if the user wishes to pan across the image to bring another part of it into view, then, once the pan function has been selected using a mode change gesture, the operator can say the words "lat", or "latitude", followed by a reference number, and do the same for longitude, and the speech recognition software would convert this data into digital form. This would then be sent to the image manipulation software which would display the correct part of the image. As one alternative to this, the user, after selecting pan mode, could say the word "bookmark" followed by the name of a previously stored bookmark, as described above, and the image would then pan to the area assigned to that bookmark.

In this way, the number of gestures that the user is required to learn is reduced, as the required action is chosen by a combination of gesture and voice, not just gesture alone. Of course, many functions or sub functions of the software can be controlled in such a manner, not just the ones referred to above. The addition of speech adds an even more natural way of interfacing with the computer system and software than just gesture alone.

There are several suppliers of speech recognition technology for computer systems that are suitable for implementing the current invention. The ViaVoice system from IBM consists of a speech recognition program that takes it's input from the sound interface card found in most modern computer systems. Dragon Systems also produce speech recognition systems that can be used to implement the speech interface of the current invention.

As another aspect of the invention, the speech recognition system is used to generate written reports on the image displayed by the computer. An example of this is now described. After selecting an area using a mode operation gesture as described above, the user draws a mode control gesture pre-programmed to invoke the report generator. As an area has been selected, the report generator will know that the report is to cover just that selected area. It will note the co-ordinates of the area and store these alongside the report itself. Once the report generator has been invoked, it will record all speech input in the report until a gesture is given indicating that the report is complete. If an area had not been selected before the report generator had been invoked, then the software would know that the report is to apply to the whole image. As another aspect of the invention, the mode change operation can be performed by means of touching menu options that can be shown on the display. To avoid cluttering up the display with such items, it is an aim of the current invention to minimise the number of items that will interfere with the image being viewed. To achieve this, the menu system operates thus: Initially, the main options, such as move, copy, zoom etc are shown, as buttons on the display. When one of these is chosen, the buttons are removed from the display, and replaced by buttons representing any sub options for the function selected, plus an option to go up the hierarchy of the menu structure. When one of these is chosen, these menu options are in turn replaced by any further options lower down in the structure. In this manner, superfluous options are not left on the display taking up space.

According to another aspect of the present invention there is provided a method of inputting information to a computer system having a position sensitive display characterised in that a user draws a gesture on the display; the gesture is digitised, and this digital representation is compared against other gestures stored in a library of gestures, where each gesture in the library is associated with a software function; the software function is carried out that is associated with the gesture in the library that matches the gesture drawn by the user.

According to another aspect of the present invention there is provided a computer program product that can be used with a computer system having a display, a position sensitive panel, and a sound digitiser, the computer program product comprising: a gesture recogniser and processor to detect and match, with pre- stored gestures stored in a gesture library, any gestures made on the position sensitive panel; a command processor that takes in commands from the gesture recogniser; image manipulation software that displays an image on the display and has manipulation functionality; whereby the gesture processor, on detecting a gesture that matches one in the library, sends a command associated with the library gesture to the command handler, which in turn sends it to the image manipulation software.

Drawings Examples of the current invention will now be described with reference to the accompanying illustrative drawings.

Figure 1 shows, in block diagrammatic form, an example of a computer system upon which the current invention can be implemented.

Figure 2 shows three examples of mode change gestures, that may be used to control program functionality.

Figure 3 shows one of the gestures of Figure 2 that has been annotated with timestamp information.

Figure 4 shows both a single, and multiple fingers being used to move an object.

Figure 5 shows multiple fingers being used to rotate an object.

Figure 6 shows the hierarchical menu structure as incorporated in the current embodiment of the invention.

Figure 7 shows a high level block diagram indicating the connectivity of the software modules used to implement the invention.

Figure 8 shows in more detail the software components that go to make up the Command Handler as shown in Figure 7 Figure 9 shows a . user drawing on the touch panel, using his finger as the pointer.

Detailed Description of the Drawings

Figure 1 shows the hardware items upon which the invention is currently implemented. A display device (liyama Prolite) 1 is connected to a processing unit 3 which additionally contains a sound processing card 4, a microphone 5 and the software to implement speech recognition (IBM ViaVoice 98) on the words spoken into the microphone 5. The display device 1 is overlaid with a thin panel that is touch sensitive, such that when it is touched by a pointer 2 of some type, it transmits the position on the display device 1 that was contacted back to the gesture recogniser software. This is in communication with software controlling the displayed image on the display device, and hence the display software will know what portion of the displayed image was nearest to the point on the display being touched.

Figure 2 provides some examples of gesture traces that may be used as mode change gestures. They can all be drawn without removing the pointer from the touch panel. Note that in practice, there would be no need for the computer to actually display the trace of the gesture on the screen, except during the.gesture training phase. Seen as drawn in the figure, there is no information present that identifies the start and end points of the gestures. However, the computer stores the set of x-y reference points that make up the gesture in an array structure, a functionally contiguous set of memory locations that hold the information. The first location holds the first point recorded, the next holds the second, and so on. By means of this, it is possible to see where the gesture started and finished. Additional information can be obtained if each point making up the gesture is given a timestamp, recording the time at which it was made. Figure 3a shows a similar gesture to that in Figure 2b, that has been annotated with simulated timestamp information. The time stamp recordings shown on Figure 3a show the point on the gesture that is drawn at the indicated time interval. Here, fixed intervals of tmtervai = 0.0625 seconds are shown. It can be seen that the spatial distance between. timestamps is shorter at the curved area of the gesture, indicating that the gesture was drawn more slowly at that point. As said above, this information can be used to distinguish between this gesture, and a similar looking one that has been drawn in a different manner. For example, a gesture that is similar in shape to Figure 3a, but where all points are half the distance apart as those shown in the diagram will be regarded as a different gesture, as it would take twice as long to draw.

Figure 4 shows a user operating the system using both single and multiple pointer mode operation gestures. Figure 4a shows a user moving an area 6 on the display 1 using a single pointer gesture. Once the area 6 is defined, and the system is in "move" mode, the user places the pointer 2 - his finger in this case - on the area 6, and moves it to where he wants the image in that area to be. The defined area 6 follows the position of his finger. The position of the area 6 after the move is shown as 6'. The movement is indicated by the arrow 7. Figure 4b shows the same operation taking place using multiple pointer mode operation gestures. These are beneficial in some instances. When moving or copying areas of the display, once the desired area has been selected, pointers placed substantially at opposite sides of the area, and then dragged to some other area of the display, will move or copy that selected area with them. Figure 4b shows someone using their thumb 2a and forefinger 2b to "grasp" onto a selected area 6. They have put their fingers at opposite sides of the selected area, and the larger pair of arrows 8 indicate the next motion is to move them in towards each other in a pinching action, to simulate the grasping of an object. This pinching action is detected by the gesture recogniser as a multi-pointer gesture, and it knows, when the pointers 2a, 2b are lifted from the touch panel 1 , that the action is not yet complete. The gesture recogniser waits for the two pointers 2a, 2b to be placed back down on the touch panel 1. When it detects this, it knows that the position touched is the destination of the move or copy command action. The thick black arrow 9 shows the movement of the area. The new position of the area 6 is indicated as 6'. The small pair of arrows 10 indicate the movement of the fingers away from the area 6', indicating the completion of the move.

Rotation of areas of the image also lends itself to the use of multi-pointer gestures. Grasping a selected area 6 as described above, followed by twisting the points on the touch panel as indicated by the arrows 11 will result in the selected area 6 rotating with the pointers. Figure 5a shows this function before the rotation, and Figure 5b shows it completed, with the area 6 having been rotated by some angle.

Figure 6 shows the hierarchical menu system, with the Main Menu box showing all top level menu selections available. Some of these options, the File, the Annotate, the Region and the Adjust options have their own subcommands that effectively sit at a layer down in the hierarchy. When one of these is chosen, using voice, gesture or the on-screen menu options, the Main Menu on-screen menu options will be replaced by the sub-commands from the chosen option. Note that each of the sub-menu options have a command to go back to the menu one level up the hierarchy.

Figure 7 is a representation of the major software modules that make up the system as currently implemented. The heart of the system is the Command Handler This takes the output of the gesture recogniser module, the speech recogniser,, and the menu items shown on the display, interprets this information, and issues the commands to the image display and manipulation (IDM) module. The IDM module then processes the displayed image according to the command received. The gesture recogniser is shown as part of the IDM module because certain gestures drawn onto the touch sensitive panel are interpreted directly by the IDM module and are not passed to the Command Handler. These gestures perform functions that interface directly to the operating system, such as opening, closing and repositioning windows on the display - functions that are not directly connected with the image manipulation software. Other gestures are passed to the Command Handler. They are separated out by the gesture recogniser according to the context in which they were entered. Once the IDM module is executed, gestures drawn within the image displayed by the module will be sent to the Command Handler. In normal use, the image displayed by the IDM module will be maximised to fill the entire display area, so in this case all gestures would be sent to the Command Handler.

It will be noticed in Figure 7 that the Command Handler also communicates with the speech recogniser and the menu display modules. This is because, according to the state the IDM module is in, some commands may not be relevant, as described above. The menu system will be redrawn on the display to reflect this. Also, depending on the IDM module state, certain speech commands may not be relevant, so the speech recognition process can be made more reliable by cutting down on the vocabulary active at any one point. This means that each word or phrase input to the speech recogniser needs be compared against fewer reference words, leading to an improved recognition rate.

Figure 8 shows in greater detail the elements that make up the Command Handler. Within the command handler box in the Figure are shown five local handler units and a command demultiplexer. It is irrelevant to the local handler units how a particular command is input to the system. Once a command is received by a local handler unit, that command is examined as to whether, judged from the current state of the system, that command is appropriate. For example, say the last command to be received was the File Menu. The next valid command can only be one of Up, Open or Exit, as seen from Figure 6. If any other command is received, say by voice or gesture, the Menu local handler unit will ignore it. Each local handler knows what commands are valid at any one time. If a valid command is received, it will be sent to the IDM module to be processed.

It will be noted that the gesture and speech recogniser outputs go via the command demultiplexer, whereas the menu inputs do not. The gesture and speech recognisers send all validly received commands to the demultiplexer, which then examines each one and distributes it to the appropriate local handler unit. Each local handler unit, on the other hand, is responsible for deciding which menus are to be displayed, and so only displays items that are to be handled by it

Figure 9 shows a user manipulating an image using a computer equipped to run the current invention. A typical session might go as follows. Once the computer hardware and software is fully set up and working, the user will call up an image that he wishes to analyse, which will be shown on the screen. He may then wish to zoom in on a particular area, to examine it more closely. To do this, he needs to switch the operating mode to "zoom". Without taking his eye from the area of interest, he then draws the mode change gesture on any part of the screen to select the zoom function. The operator may then place his finger on the screen and, maintaining contact, slide his finger either up or down. Moving down will decrease magnification, and moving up will increase magnification. The operator may also speak the zoom factor required, if he knows the exact zoom ratio required. Thus, whilst in zoom mode he may say "200%", and the system will know to expand the current visible image to a size of 200% of the original.

The currently implemented system is set up so that once a command has been executed, such as the zoom mode described above, the image manipulation software switched back to pan mode. This can be regarded as a "standby" mode. This is done because the application to which the invention is currently applied is most often used in pan mode. Thus it is more convenient for the user if this mode is entered after each command. Other applications of the invention may have another mode that is entered as the "standby" mode, or they may simply remain in a given mode until the user changes the mode to another.

When the image is displayed at the correct magnification, the operator may wish to pan across to another area of the image. As stated in the above paragraph, in the current implementation pan mode will be entered automatically when the zoom function is done. The image may then be panned by placing the finger on the screen and moving it. The software detects this movement and moves the image to try to keep the same image point underneath the operator's finger. To switch from pan mode to another mode, the user then touches one of the command options shown on the display. If the menu of command options is not currently displayed, then speaking the word "menu" will show them.

An alternative method of switching out of pan mode is to use a mode change gesture. Of course, there must be a way of differentiating between a mode change gesture and the act of panning itself. One way of doing this is as follows. The mode change gesture will be done quickly and decisively, and the operator's finger will be immediately released from the screen when it has been completed. A pan action, even if it coincides with the shape of a mode change gesture will still be interpreted as a pan action if the operator draws the gesture in a manner differently to how he draws the mode change gesture - if he draws it much slower, or pauses half way through drawing it, for example. Analysis of the gesture's temporal parameters provides the information needed to make this decision.

Claims

Claims
1. A multimedia human computer interface that is useable on a computer that has installed software for enabling an operator the ability to directly manipulate images, comprising at least a position sensitive panel and a display, characterised in that one or more gestures applied anywhere on the panel are interpreted by gesture processing software according to the gesture made and are used to control operating modes of the image manipulation software.
2. A multimedia human computer interface as claimed in claim 1 where the position sensitive panel is applied to the display.
3. A multimedia human computer interface as claimed in claims 1 or 2 whereby the computer has installed a speech recognition system that comprises a microphone, a digitiser, and speech recognition software that is in communication with the gesture processing software and image manipulation software.
4. A multimedia human computer interface as claimed in claim 3 whereby a phrase of one or more words recognised by the speech recogniser is interpreted according to the phrase recognised and is useable to control operating modes of the image manipulation software.
5. A multimedia human computer interface as claimed in claim 3 whereby following a given gesture the output of the speech recogniser is storable in the computer system.
6. A multimedia human computer interface as claimed in claims 1 to 5, whereby a library of reference gestures is stored in the computer, and an operating mode of the software is linked to each gesture.
7. A multimedia human computer interface as claimed in claim 6 whereby a time profile map s producible for each gesture in the library, showing the time taken to draw each part of the gesture.
8. A multimedia human computer interface as claimed in claim 6 or claim 7 whereby a gesture input to the system is compared to each reference gesture stored in the library and if a match is found to one of these, then it is regarded by the software as equivalent.
9 A multimedia human computer interface as claimed in claim 8 whereby a gesture is only regarded as a match to a reference gesture if it is drawn at a rate substantially similar to that of the reference gesture.
10. A multimedia human computer interface as claimed in claim 8 or claim 9 whereby when a gesture is input that matches a reference gesture, the operating mode linked to that reference gesture is executable.
11. A multimedia human computer interface as claimed in claims 6 to 10 whereby the operator can change the gesture, stored in the gesture library, that is linked to each operating mode.
12. A multimedia human computer interface as claimed in claims 6 to 11 whereby a training function allows the user to add a new gesture to the library and associate with this gesture an operating mode of the image manipulation software
13. A method of inputting information to a computer system having a position sensitive display characterised in that a user draws a gesture on the display; the gesture is digitised, and this digital representation is compared against other gestures stored in a library of gestures, where each gesture in the library is associated with a software function; the software function is carried out that is associated with the gesture in the library that matches the gesture drawn by the user.
14. A method as claimed in claim 13 where the computer also incorporates a speech recognition unit whereby, the speech recognition unit takes spoken words input using a microphone connected to the system; the words recognised are compared to words stored in a library, where each word in the library is associated with a software function; the software function is carried out that is associated with the word in the library that matches the word spoken by the user.
15. A computer program product that can be used with a computer system having a display, a position sensitive panel, and a sound digitiser, the computer program product comprising: a gesture recogniser and processor to detect and match, with pre- stored gestures stored in a gesture library, any gestures made on the position sensitive panel; a command processor that takes in commands from the gesture recogniser; image manipulation software that displays an image on the display and has manipulation functionality; whereby the gesture processor, on detecting a gesture that matches one in the library, sends a command associated with the library gesture to the command .handler, which in turn sends it to the image manipulation software.
16. A computer program product as described in claim 15, whereby the computer system also incorporates a speech recognition system that interprets spoken commands from a user and sends these commands to the command processor.
17. A computer program product for operating a method according to claims 13 and 14.
18. A multimedia human computer interface and method substantially as hereinbefore described, with reference to the accompanying drawings.
PCT/GB2001/003153 2000-07-21 2001-07-19 Human-computer interface WO2002008881A2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
GB0017793.1 2000-07-21
GB0017793A GB0017793D0 (en) 2000-07-21 2000-07-21 Human computer interface

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
AU2001270808A AU2001270808A1 (en) 2000-07-21 2001-07-19 Human-computer interface
AU7080801A AU7080801A (en) 2000-07-21 2001-07-24 Human-computer interface

Publications (2)

Publication Number Publication Date
WO2002008881A2 true WO2002008881A2 (en) 2002-01-31
WO2002008881A3 WO2002008881A3 (en) 2002-08-22

Family

ID=9896004

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2001/003153 WO2002008881A2 (en) 2000-07-21 2001-07-19 Human-computer interface

Country Status (3)

Country Link
AU (2) AU2001270808A1 (en)
GB (1) GB0017793D0 (en)
WO (1) WO2002008881A2 (en)

Cited By (69)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE10244900A1 (en) * 2002-09-26 2004-04-15 Siemens Ag A method for the automatic recognition of user commands on a user interface configured as a touch screen
WO2004107156A2 (en) * 2003-05-27 2004-12-09 Koninklijke Philips Electronics N.V. Diagnostic imaging system control with multiple control functions
EP1517228A2 (en) * 2003-09-16 2005-03-23 Smart Technologies, Inc. Gesture recognition method and touch system incorporating the same
WO2006023259A2 (en) 2004-08-16 2006-03-02 Maw Wai-Lin Virtual keypad input device
WO2006066742A1 (en) * 2004-12-21 2006-06-29 Daimlerchrysler Ag Control system for a vehicle
EP1770489A2 (en) * 2005-09-26 2007-04-04 Samsung Electronics Co., Ltd. Data control method using mouse functions in a wireless terminal
DE102006009291A1 (en) * 2006-03-01 2007-09-06 Audi Ag Method and apparatus for operating at least two functional components of a system, in particular of a vehicle
WO2007133483A1 (en) 2006-05-12 2007-11-22 Microsoft Corporation Multi-touch uses, gestures, and implementation
WO2007135536A2 (en) * 2006-05-23 2007-11-29 Nokia Corporation Improved portable electronic apparatus and associated method
WO2008030976A2 (en) * 2006-09-06 2008-03-13 Apple Inc. Touch screen device, method, and graphical user interface for determining commands by applying heuristics
EP1983416A1 (en) 2007-04-20 2008-10-22 LG Electronics Inc. Editing of data using mobile communication terminal
GB2451646A (en) * 2007-08-07 2009-02-11 Johnson Electric Sa Touchless control system
WO2009088808A2 (en) * 2007-12-31 2009-07-16 Motorola, Inc. A hand-held device and method for operating a single pointer touch sensitive user interface
WO2009104062A2 (en) * 2008-02-18 2009-08-27 Sony Ericsson Mobile Communications Ab Selecting a layout
WO2009111469A2 (en) 2008-03-04 2009-09-11 Apple Inc. Touch event model programming interface
DE102008027954A1 (en) * 2008-04-25 2009-11-26 BenQ Corp., Neihu Dialogue performing method for use between touch-screen and e.g. electronic device, involves observing contact on touch-screen of electronic device, and increasing encircled region of picture on touch-screen if contact is circular track
WO2009069049A3 (en) * 2007-11-28 2009-11-26 Koninklijke Philips Electronics N.V. Sensing device and method
WO2010027803A1 (en) * 2008-08-27 2010-03-11 Apple Inc. Omnidirectional gesture detection
WO2010029506A1 (en) * 2008-09-12 2010-03-18 Koninklijke Philips Electronics N.V. Navigating in graphical user interface on handheld devices
EP2166436A1 (en) * 2008-09-12 2010-03-24 Samsung Electronics Co., Ltd. Proximity sensor based input system and method for operating the same
WO2010049877A1 (en) * 2008-10-27 2010-05-06 Nokia Corporation Methods and apparatuses for facilitating interaction with touch screen apparatuses
WO2010142543A1 (en) * 2009-06-12 2010-12-16 Volkswagen Ag Method for controlling a graphical user interface and operating device for a graphical user interface
WO2011025642A1 (en) * 2009-08-31 2011-03-03 Qualcomm Incorporated User interface methods providing searching functionality
WO2011028944A1 (en) 2009-09-02 2011-03-10 Amazon Technologies, Inc. Touch-screen user interface
EP2306288A1 (en) 2009-09-25 2011-04-06 Research In Motion Limited Electronic device including touch-sensitive input device and method of controlling same
EP1645945A3 (en) * 2004-10-05 2011-05-25 Sony Corporation Information-processing apparatus and programs used in information-processing apparatus
JP2011227828A (en) * 2010-04-22 2011-11-10 Canon Inc Information processor, information processing method and information processing program
US8265688B2 (en) 2007-12-31 2012-09-11 Motorola Mobility Llc Wireless communication device and split touch sensitive user input surface
US8285499B2 (en) 2009-03-16 2012-10-09 Apple Inc. Event recognition
US8370764B2 (en) 2004-08-30 2013-02-05 Microsoft Corporation Scrolling web pages using direct interaction
EP2573662A1 (en) * 2011-09-23 2013-03-27 Samsung Electronics Co., Ltd. Apparatus and method for controlling display size in portable terminal
US8411061B2 (en) 2008-03-04 2013-04-02 Apple Inc. Touch event processing for documents
EP2214090A3 (en) * 2009-01-28 2013-06-05 Sony Corporation Information processing apparatus, animation method, and program
US8487938B2 (en) 2009-01-30 2013-07-16 Microsoft Corporation Standard Gestures
US8493364B2 (en) 2009-04-30 2013-07-23 Motorola Mobility Llc Dual sided transparent display module and portable electronic device incorporating the same
US8552999B2 (en) 2010-06-14 2013-10-08 Apple Inc. Control selection approximation
US8566717B2 (en) 2008-06-24 2013-10-22 Microsoft Corporation Rendering teaching animations on a user-interface display
US8566045B2 (en) 2009-03-16 2013-10-22 Apple Inc. Event recognition
US8566044B2 (en) 2009-03-16 2013-10-22 Apple Inc. Event recognition
EP2674845A1 (en) * 2012-06-14 2013-12-18 ICT Automatisering N.V. User interaction via a touch screen
US8645827B2 (en) 2008-03-04 2014-02-04 Apple Inc. Touch event model
CN103959211A (en) * 2012-05-21 2014-07-30 宇龙计算机通信科技(深圳)有限公司 Terminal and switching method of application function interface
US9037995B2 (en) 2007-01-07 2015-05-19 Apple Inc. Application programming interfaces for scrolling operations
CN104756184A (en) * 2012-08-30 2015-07-01 谷歌公司 Techniques for selecting languages for automatic speech recognition
EP2513760A4 (en) * 2009-12-18 2016-01-06 Synaptics Inc Method and apparatus for changing operating modes
US9311112B2 (en) 2009-03-16 2016-04-12 Apple Inc. Event recognition
US9372620B2 (en) 2007-01-07 2016-06-21 Apple Inc. Portable multifunction device, method, and graphical user interface for interacting with user input elements in displayed content
US9384672B1 (en) 2006-03-29 2016-07-05 Amazon Technologies, Inc. Handheld electronic book reader device having asymmetrical shape
US9448711B2 (en) 2005-05-23 2016-09-20 Nokia Technologies Oy Mobile communication terminal and associated methods
US9465532B2 (en) 2009-12-18 2016-10-11 Synaptics Incorporated Method and apparatus for operating in pointing and enhanced gesturing modes
US9524094B2 (en) 2009-02-20 2016-12-20 Nokia Technologies Oy Method and apparatus for causing display of a cursor
US9529519B2 (en) 2007-01-07 2016-12-27 Apple Inc. Application programming interfaces for gesture operations
US9619132B2 (en) 2007-01-07 2017-04-11 Apple Inc. Device, method and graphical user interface for zooming in on a touch-screen display
US9619143B2 (en) 2008-01-06 2017-04-11 Apple Inc. Device, method, and graphical user interface for viewing application launch icons
US9626073B2 (en) 2002-03-19 2017-04-18 Facebook, Inc. Display navigation
US9684521B2 (en) 2010-01-26 2017-06-20 Apple Inc. Systems having discrete and continuous gesture recognizers
RU2623198C2 (en) * 2011-08-02 2017-06-27 МАЙКРОСОФТ ТЕКНОЛОДЖИ ЛАЙСЕНСИНГ, ЭлЭлСи Diagonal sliding gesture for selection and permutation
US9733716B2 (en) 2013-06-09 2017-08-15 Apple Inc. Proxy gesture recognizer
US9733812B2 (en) 2010-01-06 2017-08-15 Apple Inc. Device, method, and graphical user interface with content display modes and display rotation heuristics
US9772751B2 (en) 2007-06-29 2017-09-26 Apple Inc. Using gestures to slide between user interfaces
US9798459B2 (en) 2008-03-04 2017-10-24 Apple Inc. Touch event model for web pages
CN103809908B (en) * 2008-03-04 2018-02-09 苹果公司 Touch event model Programming Interface
US9933937B2 (en) 2007-06-20 2018-04-03 Apple Inc. Portable multifunction device, method, and graphical user interface for playing online videos
US9933913B2 (en) 2005-12-30 2018-04-03 Apple Inc. Portable electronic device with interface reconfiguration mode
US9977518B2 (en) 2001-10-22 2018-05-22 Apple Inc. Scrolling based on rotational movement
US10139870B2 (en) 2006-07-06 2018-11-27 Apple Inc. Capacitance sensing electrode with integrated I/O mechanism
US10180732B2 (en) 2006-10-11 2019-01-15 Apple Inc. Gimballed scroll wheel
US10254949B2 (en) 2007-01-07 2019-04-09 Apple Inc. Portable multifunction device, method, and graphical user interface supporting user navigations of graphical objects on a touch screen display
US10272294B2 (en) 2016-06-11 2019-04-30 Apple Inc. Activity and workout updates

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070152983A1 (en) 2005-12-30 2007-07-05 Apple Computer, Inc. Touch pad with symbols based on mode
US7748634B1 (en) 2006-03-29 2010-07-06 Amazon Technologies, Inc. Handheld electronic book reader device having dual displays
US8059099B2 (en) 2006-06-02 2011-11-15 Apple Inc. Techniques for interactive input to portable electronic devices
US9360967B2 (en) 2006-07-06 2016-06-07 Apple Inc. Mutual capacitance touch sensing device
US8743060B2 (en) 2006-07-06 2014-06-03 Apple Inc. Mutual capacitance touch sensing device
US8416198B2 (en) 2007-12-03 2013-04-09 Apple Inc. Multi-dimensional scroll wheel
US9454256B2 (en) 2008-03-14 2016-09-27 Apple Inc. Sensor configurations of an input device that are switchable based on mode
US9354751B2 (en) 2009-05-15 2016-05-31 Apple Inc. Input device with optimized capacitive sensing
US8872771B2 (en) 2009-07-07 2014-10-28 Apple Inc. Touch sensing device having conductive nodes
US8451238B2 (en) 2009-09-02 2013-05-28 Amazon Technologies, Inc. Touch-screen user interface
US8949735B2 (en) 2012-11-02 2015-02-03 Google Inc. Determining scroll direction intent

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5252951A (en) * 1989-04-28 1993-10-12 International Business Machines Corporation Graphical user interface with gesture recognition in a multiapplication environment
US5677710A (en) * 1993-05-10 1997-10-14 Apple Computer, Inc. Recognition keypad
US5864635A (en) * 1996-06-14 1999-01-26 International Business Machines Corporation Distinguishing gestures from handwriting in a pen based computer by stroke analysis

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5252951A (en) * 1989-04-28 1993-10-12 International Business Machines Corporation Graphical user interface with gesture recognition in a multiapplication environment
US5677710A (en) * 1993-05-10 1997-10-14 Apple Computer, Inc. Recognition keypad
US5864635A (en) * 1996-06-14 1999-01-26 International Business Machines Corporation Distinguishing gestures from handwriting in a pen based computer by stroke analysis

Cited By (143)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9977518B2 (en) 2001-10-22 2018-05-22 Apple Inc. Scrolling based on rotational movement
US9626073B2 (en) 2002-03-19 2017-04-18 Facebook, Inc. Display navigation
US9886163B2 (en) 2002-03-19 2018-02-06 Facebook, Inc. Constrained display navigation
US10055090B2 (en) 2002-03-19 2018-08-21 Facebook, Inc. Constraining display motion in display navigation
US9753606B2 (en) 2002-03-19 2017-09-05 Facebook, Inc. Animated display navigation
US9678621B2 (en) 2002-03-19 2017-06-13 Facebook, Inc. Constraining display motion in display navigation
US9851864B2 (en) 2002-03-19 2017-12-26 Facebook, Inc. Constraining display in display navigation
EP1408395A3 (en) * 2002-09-26 2015-11-25 Siemens Aktiengesellschaft Method for automatic identification of user commands on a touch screen user interface
DE10244900A1 (en) * 2002-09-26 2004-04-15 Siemens Ag A method for the automatic recognition of user commands on a user interface configured as a touch screen
WO2004107156A3 (en) * 2003-05-27 2005-04-28 Koninkl Philips Electronics Nv Diagnostic imaging system control with multiple control functions
WO2004107156A2 (en) * 2003-05-27 2004-12-09 Koninklijke Philips Electronics N.V. Diagnostic imaging system control with multiple control functions
EP1517228A2 (en) * 2003-09-16 2005-03-23 Smart Technologies, Inc. Gesture recognition method and touch system incorporating the same
EP1517228A3 (en) * 2003-09-16 2008-04-02 Smart Technologies, Inc. Gesture recognition method and touch system incorporating the same
EP2270630A3 (en) * 2003-09-16 2011-03-02 SMART Technologies ULC Gesture recognition method and touch system incorporating the same
EP1779373A2 (en) * 2004-08-16 2007-05-02 Maw Wai-Lin Virtual keypad input device
EP1779373A4 (en) * 2004-08-16 2011-07-13 Maw Wai-Lin Virtual keypad input device
US8797192B2 (en) 2004-08-16 2014-08-05 Wai-Lin Maw Virtual keypad input device
WO2006023259A2 (en) 2004-08-16 2006-03-02 Maw Wai-Lin Virtual keypad input device
US8370764B2 (en) 2004-08-30 2013-02-05 Microsoft Corporation Scrolling web pages using direct interaction
US9342232B2 (en) 2004-10-05 2016-05-17 Sony Corporation Information-processing apparatus providing multiple display modes
EP1645945A3 (en) * 2004-10-05 2011-05-25 Sony Corporation Information-processing apparatus and programs used in information-processing apparatus
US9052813B2 (en) 2004-10-05 2015-06-09 Sony Corporation Information-processing apparatus and programs used in information-processing apparatus
WO2006066742A1 (en) * 2004-12-21 2006-06-29 Daimlerchrysler Ag Control system for a vehicle
US9785329B2 (en) 2005-05-23 2017-10-10 Nokia Technologies Oy Pocket computer and associated methods
US9448711B2 (en) 2005-05-23 2016-09-20 Nokia Technologies Oy Mobile communication terminal and associated methods
EP1770489A2 (en) * 2005-09-26 2007-04-04 Samsung Electronics Co., Ltd. Data control method using mouse functions in a wireless terminal
EP1770489A3 (en) * 2005-09-26 2012-09-19 Samsung Electronics Co., Ltd. Data control method using mouse functions in a wireless terminal
US9933913B2 (en) 2005-12-30 2018-04-03 Apple Inc. Portable electronic device with interface reconfiguration mode
DE102006009291A1 (en) * 2006-03-01 2007-09-06 Audi Ag Method and apparatus for operating at least two functional components of a system, in particular of a vehicle
US9384672B1 (en) 2006-03-29 2016-07-05 Amazon Technologies, Inc. Handheld electronic book reader device having asymmetrical shape
EP2027525A1 (en) * 2006-05-12 2009-02-25 Microsoft Corporation Multi-touch uses, gestures, and implementation
EP2027525A4 (en) * 2006-05-12 2012-01-04 Microsoft Corp Multi-touch uses, gestures, and implementation
US9996176B2 (en) 2006-05-12 2018-06-12 Microsoft Technology Licensing, Llc Multi-touch uses, gestures, and implementation
US9063647B2 (en) 2006-05-12 2015-06-23 Microsoft Technology Licensing, Llc Multi-touch uses, gestures, and implementation
US9811186B2 (en) 2006-05-12 2017-11-07 Microsoft Technology Licensing, Llc Multi-touch uses, gestures, and implementation
WO2007133483A1 (en) 2006-05-12 2007-11-22 Microsoft Corporation Multi-touch uses, gestures, and implementation
WO2007135536A2 (en) * 2006-05-23 2007-11-29 Nokia Corporation Improved portable electronic apparatus and associated method
WO2007135536A3 (en) * 2006-05-23 2008-08-21 Nokia Corp Improved portable electronic apparatus and associated method
US10139870B2 (en) 2006-07-06 2018-11-27 Apple Inc. Capacitance sensing electrode with integrated I/O mechanism
US9335924B2 (en) 2006-09-06 2016-05-10 Apple Inc. Touch screen device, method, and graphical user interface for customizing display of content category icons
US9952759B2 (en) 2006-09-06 2018-04-24 Apple Inc. Touch screen device, method, and graphical user interface for customizing display of content category icons
CN101861562B (en) * 2006-09-06 2016-05-25 苹果公司 Command is determined by applying the heuristics touch screen devices, methods, and graphical user interface
KR101462363B1 (en) * 2006-09-06 2014-11-17 애플 인크. Touch screen device, method, and graphical user interface for determining commands by applying heuristics
WO2008030976A3 (en) * 2006-09-06 2009-11-26 Apple Inc. Touch screen device, method, and graphical user interface for determining commands by applying heuristics
WO2008030976A2 (en) * 2006-09-06 2008-03-13 Apple Inc. Touch screen device, method, and graphical user interface for determining commands by applying heuristics
KR101476019B1 (en) * 2006-09-06 2014-12-23 애플 인크. Touch screen device, method, and graphical user interface for determining commands by applying heuristics
US10180732B2 (en) 2006-10-11 2019-01-15 Apple Inc. Gimballed scroll wheel
US9575648B2 (en) 2007-01-07 2017-02-21 Apple Inc. Application programming interfaces for gesture operations
US9619132B2 (en) 2007-01-07 2017-04-11 Apple Inc. Device, method and graphical user interface for zooming in on a touch-screen display
US9639260B2 (en) 2007-01-07 2017-05-02 Apple Inc. Application programming interfaces for gesture operations
US9037995B2 (en) 2007-01-07 2015-05-19 Apple Inc. Application programming interfaces for scrolling operations
US9665265B2 (en) 2007-01-07 2017-05-30 Apple Inc. Application programming interfaces for gesture operations
US10175876B2 (en) 2007-01-07 2019-01-08 Apple Inc. Application programming interfaces for gesture operations
US9760272B2 (en) 2007-01-07 2017-09-12 Apple Inc. Application programming interfaces for scrolling operations
US10228824B2 (en) 2007-01-07 2019-03-12 Apple Inc. Portable multifunction device, method, and graphical user interface for interacting with user input elements in displayed content
US9448712B2 (en) 2007-01-07 2016-09-20 Apple Inc. Application programming interfaces for scrolling operations
US10254949B2 (en) 2007-01-07 2019-04-09 Apple Inc. Portable multifunction device, method, and graphical user interface supporting user navigations of graphical objects on a touch screen display
US9529519B2 (en) 2007-01-07 2016-12-27 Apple Inc. Application programming interfaces for gesture operations
US9372620B2 (en) 2007-01-07 2016-06-21 Apple Inc. Portable multifunction device, method, and graphical user interface for interacting with user input elements in displayed content
US8856689B2 (en) 2007-04-20 2014-10-07 Lg Electronics Inc. Editing of data using mobile communication terminal
EP1983416A1 (en) 2007-04-20 2008-10-22 LG Electronics Inc. Editing of data using mobile communication terminal
US9933937B2 (en) 2007-06-20 2018-04-03 Apple Inc. Portable multifunction device, method, and graphical user interface for playing online videos
US9772751B2 (en) 2007-06-29 2017-09-26 Apple Inc. Using gestures to slide between user interfaces
GB2451646A (en) * 2007-08-07 2009-02-11 Johnson Electric Sa Touchless control system
WO2009069049A3 (en) * 2007-11-28 2009-11-26 Koninklijke Philips Electronics N.V. Sensing device and method
EP2752738A3 (en) * 2007-11-28 2017-02-15 Koninklijke Philips N.V. Sensing device and method
US8525805B2 (en) 2007-11-28 2013-09-03 Koninklijke Philips N.V. Sensing device and method
WO2009088808A3 (en) * 2007-12-31 2009-11-26 Motorola, Inc. A hand-held device and method for operating a single pointer touch sensitive user interface
US8265688B2 (en) 2007-12-31 2012-09-11 Motorola Mobility Llc Wireless communication device and split touch sensitive user input surface
RU2503989C2 (en) * 2007-12-31 2014-01-10 Моторола Мобилити, Инк. Portable device and method of operating single-pointer touch-sensitive user interface
US8707215B2 (en) 2007-12-31 2014-04-22 Motorola Mobility Llc Hand-held device and method for operating a single pointer touch sensitive user interface
KR101217934B1 (en) * 2007-12-31 2013-01-02 모토로라 모빌리티 엘엘씨 A method and handheld device for operating a single pointer touch sensitive user interface,
WO2009088808A2 (en) * 2007-12-31 2009-07-16 Motorola, Inc. A hand-held device and method for operating a single pointer touch sensitive user interface
US9619143B2 (en) 2008-01-06 2017-04-11 Apple Inc. Device, method, and graphical user interface for viewing application launch icons
WO2009104062A3 (en) * 2008-02-18 2009-11-26 Sony Ericsson Mobile Communications Ab Selecting a layout
WO2009104062A2 (en) * 2008-02-18 2009-08-27 Sony Ericsson Mobile Communications Ab Selecting a layout
EP2405343A3 (en) * 2008-03-04 2012-02-15 Apple Inc. Touch event model programming interface
US8836652B2 (en) 2008-03-04 2014-09-16 Apple Inc. Touch event model programming interface
US8645827B2 (en) 2008-03-04 2014-02-04 Apple Inc. Touch event model
EP2405346A3 (en) * 2008-03-04 2012-02-15 Apple Inc. Touch event model programming interface
WO2009111469A2 (en) 2008-03-04 2009-09-11 Apple Inc. Touch event model programming interface
US8723822B2 (en) 2008-03-04 2014-05-13 Apple Inc. Touch event model programming interface
EP2405345A3 (en) * 2008-03-04 2012-02-15 Apple Inc. Touch event model programming interface
CN103761044A (en) * 2008-03-04 2014-04-30 苹果公司 Touch event model programming interface
EP2405344A3 (en) * 2008-03-04 2012-02-15 Apple Inc. Touch event model programming interface
US9798459B2 (en) 2008-03-04 2017-10-24 Apple Inc. Touch event model for web pages
CN103809908B (en) * 2008-03-04 2018-02-09 苹果公司 Touch event model Programming Interface
CN103809908A (en) * 2008-03-04 2014-05-21 苹果公司 Touch Event Model Programming Interface
WO2009111469A3 (en) * 2008-03-04 2010-01-14 Apple Inc. Touch event model programming interface
US9720594B2 (en) 2008-03-04 2017-08-01 Apple Inc. Touch event model
US9971502B2 (en) 2008-03-04 2018-05-15 Apple Inc. Touch event model
US8411061B2 (en) 2008-03-04 2013-04-02 Apple Inc. Touch event processing for documents
US9690481B2 (en) 2008-03-04 2017-06-27 Apple Inc. Touch event model
JP2010521037A (en) * 2008-03-04 2010-06-17 アップル インコーポレイテッド Touch event model programming interface
DE102008027954A1 (en) * 2008-04-25 2009-11-26 BenQ Corp., Neihu Dialogue performing method for use between touch-screen and e.g. electronic device, involves observing contact on touch-screen of electronic device, and increasing encircled region of picture on touch-screen if contact is circular track
US8566717B2 (en) 2008-06-24 2013-10-22 Microsoft Corporation Rendering teaching animations on a user-interface display
WO2010027803A1 (en) * 2008-08-27 2010-03-11 Apple Inc. Omnidirectional gesture detection
EP2166436A1 (en) * 2008-09-12 2010-03-24 Samsung Electronics Co., Ltd. Proximity sensor based input system and method for operating the same
JP2012502386A (en) * 2008-09-12 2012-01-26 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Navigation in the graphical user interface of the portable device
US8645858B2 (en) 2008-09-12 2014-02-04 Koninklijke Philips N.V. Navigating in graphical user interface on handheld devices
CN102150160B (en) * 2008-09-12 2015-05-20 皇家飞利浦电子股份有限公司 Navigating in graphical user interface on handheld devices
WO2010029506A1 (en) * 2008-09-12 2010-03-18 Koninklijke Philips Electronics N.V. Navigating in graphical user interface on handheld devices
KR101611601B1 (en) * 2008-09-12 2016-04-12 코닌클리케 필립스 엔.브이. Navigating in graphical user interface on handheld devices
CN102150160A (en) * 2008-09-12 2011-08-10 皇家飞利浦电子股份有限公司 Navigating in graphical user interface on handheld devices
WO2010049877A1 (en) * 2008-10-27 2010-05-06 Nokia Corporation Methods and apparatuses for facilitating interaction with touch screen apparatuses
EP2214090A3 (en) * 2009-01-28 2013-06-05 Sony Corporation Information processing apparatus, animation method, and program
US8487938B2 (en) 2009-01-30 2013-07-16 Microsoft Corporation Standard Gestures
US9524094B2 (en) 2009-02-20 2016-12-20 Nokia Technologies Oy Method and apparatus for causing display of a cursor
US8285499B2 (en) 2009-03-16 2012-10-09 Apple Inc. Event recognition
US8566045B2 (en) 2009-03-16 2013-10-22 Apple Inc. Event recognition
US9965177B2 (en) 2009-03-16 2018-05-08 Apple Inc. Event recognition
US8566044B2 (en) 2009-03-16 2013-10-22 Apple Inc. Event recognition
US9311112B2 (en) 2009-03-16 2016-04-12 Apple Inc. Event recognition
US8428893B2 (en) 2009-03-16 2013-04-23 Apple Inc. Event recognition
US9483121B2 (en) 2009-03-16 2016-11-01 Apple Inc. Event recognition
US9285908B2 (en) 2009-03-16 2016-03-15 Apple Inc. Event recognition
US8493364B2 (en) 2009-04-30 2013-07-23 Motorola Mobility Llc Dual sided transparent display module and portable electronic device incorporating the same
WO2010142543A1 (en) * 2009-06-12 2010-12-16 Volkswagen Ag Method for controlling a graphical user interface and operating device for a graphical user interface
US8910086B2 (en) 2009-06-12 2014-12-09 Volkswagen Ag Method for controlling a graphical user interface and operating device for a graphical user interface
WO2011025642A1 (en) * 2009-08-31 2011-03-03 Qualcomm Incorporated User interface methods providing searching functionality
CN102483679A (en) * 2009-08-31 2012-05-30 高通股份有限公司 User interface methods providing searching functionality
WO2011028944A1 (en) 2009-09-02 2011-03-10 Amazon Technologies, Inc. Touch-screen user interface
EP2473897A4 (en) * 2009-09-02 2013-01-23 Amazon Tech Inc Touch-screen user interface
KR20120073223A (en) * 2009-09-02 2012-07-04 아마존 테크놀로지스, 인크. Touch-screen user interface
KR101675178B1 (en) * 2009-09-02 2016-11-10 아마존 테크놀로지스, 인크. The touch-screen user interface
EP2473897A1 (en) * 2009-09-02 2012-07-11 Amazon Technologies, Inc. Touch-screen user interface
EP2306288A1 (en) 2009-09-25 2011-04-06 Research In Motion Limited Electronic device including touch-sensitive input device and method of controlling same
US9465532B2 (en) 2009-12-18 2016-10-11 Synaptics Incorporated Method and apparatus for operating in pointing and enhanced gesturing modes
EP2513760A4 (en) * 2009-12-18 2016-01-06 Synaptics Inc Method and apparatus for changing operating modes
US9733812B2 (en) 2010-01-06 2017-08-15 Apple Inc. Device, method, and graphical user interface with content display modes and display rotation heuristics
US9684521B2 (en) 2010-01-26 2017-06-20 Apple Inc. Systems having discrete and continuous gesture recognizers
JP2011227828A (en) * 2010-04-22 2011-11-10 Canon Inc Information processor, information processing method and information processing program
US10216408B2 (en) 2010-06-14 2019-02-26 Apple Inc. Devices and methods for identifying user interface objects based on view hierarchy
US8552999B2 (en) 2010-06-14 2013-10-08 Apple Inc. Control selection approximation
RU2623198C2 (en) * 2011-08-02 2017-06-27 МАЙКРОСОФТ ТЕКНОЛОДЖИ ЛАЙСЕНСИНГ, ЭлЭлСи Diagonal sliding gesture for selection and permutation
EP2573662A1 (en) * 2011-09-23 2013-03-27 Samsung Electronics Co., Ltd. Apparatus and method for controlling display size in portable terminal
CN103106023A (en) * 2011-09-23 2013-05-15 三星电子株式会社 Apparatus and method for controlling display size in portable terminal
US9471218B2 (en) 2011-09-23 2016-10-18 Samsung Electronics Co., Ltd. Apparatus and method for controlling display size in portable terminal
CN103959211A (en) * 2012-05-21 2014-07-30 宇龙计算机通信科技(深圳)有限公司 Terminal and switching method of application function interface
EP2674845A1 (en) * 2012-06-14 2013-12-18 ICT Automatisering N.V. User interaction via a touch screen
CN104756184A (en) * 2012-08-30 2015-07-01 谷歌公司 Techniques for selecting languages for automatic speech recognition
US9733716B2 (en) 2013-06-09 2017-08-15 Apple Inc. Proxy gesture recognizer
US10272294B2 (en) 2016-06-11 2019-04-30 Apple Inc. Activity and workout updates

Also Published As

Publication number Publication date
AU2001270808A1 (en) 2002-02-05
WO2002008881A3 (en) 2002-08-22
GB0017793D0 (en) 2000-09-06
AU7080801A (en) 2002-02-05

Similar Documents

Publication Publication Date Title
Kurtenbach The design and evaluation of marking menus.
Mynatt et al. Nonvisual presentation of graphical user interfaces: contrasting two approaches
US6232957B1 (en) Technique for implementing an on-demand tool glass for use in a desktop user interface
US6333753B1 (en) Technique for implementing an on-demand display widget through controlled fading initiated by user contact with a touch sensitive input device
EP2847658B1 (en) Device, method, and graphical user interface for providing tactile feedback for operations performed in a user interface
US9195321B2 (en) Input device user interface enhancements
JP4577428B2 (en) Display device, a display method, and program
US8881061B2 (en) Device, method, and graphical user interface for managing folders
EP2652579B1 (en) Detecting gestures involving movement of a computing device
US9152317B2 (en) Manipulation of graphical elements via gestures
EP2511812B1 (en) Continuous recognition method of multi-touch gestures from at least two multi-touch input devices
CN102625931B (en) For initiating activities in the electronic device user interface
US8239784B2 (en) Mode-based graphical user interfaces for touch sensitive input devices
US7877707B2 (en) Detecting and interpreting real-world and security gestures on touch and hover sensitive devices
AU2008100085A4 (en) Gesturing with a multipoint sensing device
EP0827064B1 (en) Selection device for touchscreen systems
EP2652580B1 (en) Using movement of a computing device to enhance interpretation of input events produced when interacting with the computing device
US7768501B1 (en) Method and system for touch screen keyboard and display space sharing
US8791900B2 (en) Computing device notes
AU2007209926B2 (en) Gesturing with a multipoint sensing device
US9996231B2 (en) Device, method, and graphical user interface for manipulating framed graphical objects
EP2526469B1 (en) Eye tracker based contextual action
KR100382100B1 (en) Computer system and method for manipulating multiple graphical user interface components on a computer display having proximity pointers
US8065624B2 (en) Virtual keypad systems and methods
EP3121697A1 (en) Mode-based graphical user interfaces for touch sensitive input devices

Legal Events

Date Code Title Description
AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG US UZ VN YU ZA ZW

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
AK Designated states

Kind code of ref document: A3

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG US UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A3

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
ENP Entry into the national phase in:

Ref document number: 2003134363

Country of ref document: RU

Kind code of ref document: A

Format of ref document f/p: F

NENP Non-entry into the national phase in:

Ref country code: JP