DE102009019910B4 - Gesture recognition - Google Patents

Abstract

A touch sensor device comprising: a sensor having a sensitive area extending in at least one dimension and arranged to output detection signals in response to the proximity of an object to the sensitive area, a position processing unit operable to determine positions of interactions with the sensitive area from an analysis of the detection signals and to output a time sequence of data indicating the interaction positions on the sensor corresponding to the respective touches, and a gesture processing unit which can be operated to analyze the time sequence of data and distinguish one or more gestures from the sequence of data, wherein the gesture processing unit is encoded with a gesture recognition code comprising a plurality of state modules of a state machine, the plurality of state modules comprising: a first single touch State module of the state machine operable to recognize at least one first single touch gesture and generate a first output based on the first single touch gesture, a second single touch state module of the state machine operable to perform at least one second single touch gesture recognize and generate a second output based on the second single-touch gesture, and a multi-touch state module of the state machine operable to receive the first output directly from the first single-touch state module to output the second output directly from the second single-touch To receive the state module, and to recognize at least one multi-touch gesture based at least on the first and second outputs, the first single-touch status module, the second single-touch status module and the multi-touch status module being distinguishable, u nd to output the recognized multiple touch gesture.

Description

The present invention relates to gesture recognition and, more particularly, to gesture recognition by processing a time sequence of position inputs received via a two-dimensional (2D) touch sensor such as a capacitive or resistive touch sensor. The invention can also be applied to one-dimensional (1D) touch sensors, and the principles can also be applied to three-dimensional sensors. In addition, the invention can be applied to proximity sensors in which no physical contact, i.e. H. there is no contact with a sensor surface. The invention can also be applied to sensor surfaces that can be operated by the finger of an operator or by a pen.

Capacitive and resistive 1D and 2D touch sensors have been in widespread use for many years. These include, for example, the touch screens of PDAs, MP3 players, cell phones or other multimedia devices. The touchpad, which is used in place of a mouse in notebook computers, is another example of a 2D capacitive touch sensor. 2D sensors are also found in many household appliances such as ovens and mixers.

2D capacitive sensors have often been described in detail, for example in patents and patent applications US 2005/0041018 A1 , US 2007/0247443 A1 , US 2007/0257894 A1 and US 2007/0279395 A1 by the inventor Harald Philipp, the entire content of which is hereby incorporated by reference.

• US 3,593,115 gibt ein Berührungselement mit einer dreieckigen Form zur Bestimmung einer Objektposition an. Für dieses Element sind jedoch zahlreiche sekundäre Elektrodenverbindungen sowie zwei oder mehr Schichten erforderlich, wodurch die Kosten erhöht werden und die Transparenz reduziert wird.

Other examples of prior art touch screens are listed below:

• U.S. 3,593,115 indicates a touch element having a triangular shape for determining an object position. However, this element requires numerous secondary electrode connections and two or more layers, thereby increasing the cost and reducing the transparency.

U.S. 5,650,597 specifies a 2D sensor method that only requires one layer in its active area, but requires a large number of electrode connections. Resistive strips resolve a position axis, the accuracy depending on the tolerance of a large number of resistive strips. However, this method suppresses hand shadow effects.

U.S. 6,297,811 describes a touch screen that uses triangular wire outline electrode shapes to create field gradients. However, this patent suffers from the problem that it is difficult to enlarge the screen size because it requires one electrode connection with one sensor circuit per triangle. It would be desirable to reduce the number of connections to reduce cost and simplify construction. It would also be desirable to use solid shapes in place of the wire outlines, which are more expensive to manufacture. However, this method does not suppress hand shadow effects.

Gesture recognition has been used in such devices for many years. An early example of this is character recognition in PDAs such as the original Palm Inc. devices. Dragging your finger and tapping once and twice on a notebook's touchpad are other long-standing examples. Recently, gesture recognition has been incorporated into handheld devices such as the Apple iPhone (registered trademark). In addition, there are many prior art patent publications on touch screens that use gesture recognition, with Synaptics, Inc. in particular and, more recently, Apple Computer, Inc. having many such publications.

US 2007/152984 A1 by Apple Computer, Inc. discloses a portable communication device having a multiple-touch input device that detects one or more touch contacts and movements and performs one or more operations on an object based on the one or more multiple contacts and / or movements.

US 2002/015024 A1 from the University of Delaware discloses tracking multiple finger and palm contacts simultaneously, such as approach, touch, and slide of a hand over a proximity sensor. Segmentation processing extracts shape, position, and surface proximity features for each contact, using a path tracker to detect individual touchdowns and elevations. Combinatorial optimization modules associate the path of each contact with a particular fingertip, thumb, or palm of a hand based on biomechanical constraints and contact features. A classification of intuitive hand configurations and movements enables a previously impossible integration of typing, resting, pointing, scrolling, 3D manipulation and handwriting into a versatile and ergonomic computer input device.

U.S. 5,852,352 indicates a touch panel that can detect multiple touches at the same time. In an xy electrode array maxima and minima are defined in each of the x and y signals, where the maxima are referred to as finger touches. The peak and valley data in the x and y directions are then interpolated to identify the position of one or more fingers on the sensor array.

US 6028271 , US 6414671 and US 6750852 are related patents owned by Synaptics, Inc. that disclose gesture recognition of an object on a touch sensor panel and for cursor movement. Typing, pulling, pushing, longer pulling and variable pulling gestures are recognized by analyzing the position, pressure and movement of the conductive object on the sensor panel during an expected gesture, with the signals being sent to a host, who is aware of the occurrence of this Indicating gestures.

US2007 / 176906 A1 by Synaptics, Inc. discloses a touch sensor that includes a signal processor configured to distinguish between three gestures based on different finger movements on the sensor device, and as in FIG 5 from US2007 / 17906 A1 comprises a workflow with an idle state and three gesture-specific states, each of which is referred to as the first, second and third result state.

In general, the raw output from the 2D touch sensor is a time sequence of x, y coordinates which are then processed by software or by firmware generated by means of higher-level software in order to distinguish the nature of the entered gesture. In general, the raw data is divided into adjacent touch segments and then processed to determine which gestures can be deduced. The processing of the raw data to identify the gestures can take place in the same chip that generates the raw data: The raw data can, however, also be exported to an external chip, for example by means of a transmission via a communication bus to the CPU of the device. The first approach is preferred by Synaptics, while the second approach is used, for example, by in US 2006/0066582 A1 used by Apple.
US 2007/046643 A1 discloses a method and apparatus for identifying a user gesture with a touch sensor to determine contact information describing locations where a user touches a touch-sensitive surface corresponding to a screen. The touch sensor provides the contact information to a gesture identification module that uses status information to identify a user gesture and, in response, issues an associated screen command to a screen control module. The screen control module updates the screen based on screen commands received from the gesture identification module.
US 2008/0165141 A1 discloses a system, method, and software for implementing gestures with touch-sensitive devices (such as a touch-sensitive display) for managing and editing media files on a computing device or system. In particular, gestural input from a human hand via a touch / proximity sensitive device can be used to control, edit and manipulate files, for example media files including graphics files, photo files and video files.
US 2007/0291009 A1 discloses an apparatus and method for distinguishing a particular gesture from multiple gestures performed by a conductive object on the sensing device using less than three time intervals. The device may include a detection device for detecting a presence of a conductive object and a processing device coupled to the detection device for distinguishing the plurality of gestures. The method can distinguish between a tap gesture, a double tap gesture, a drag gesture and a movement gesture.

The patent literature is mostly unspecific with regard to how the temporal sequence of the raw data is converted into gestures. The direct approach provides that a corresponding higher-level code is written, for example, in C or in another suitable programming language, whereby the interpretation of the temporal sequence of the data is carried out using conditional statements such as "if", "then" and " else "is analyzed.

However, it is difficult to reliably and efficiently add code to identify a new gesture in an existing block of code and to distinguish between a larger number of gestures such as 3-4 or 10-20. Testing the code presents particular difficulties. The reason for this is generally that the input at any intermediate point in a time sequence of x, y, t data can relate to a large number of possible gestures, so that the coding for recognizing a gesture generally depends on the coding for Recognition of another gesture or is linked to it.

The problem underlying the present invention is to provide a touch sensor device for recognizing the gestures and for calculating the positions of an object from interactions with the sensitive area.

The problem on which the invention is based is solved by the subject matter of the independent claims.

The present invention solves this problem by using a state machine approach to design and write the gesture recognition algorithm. More particularly, the invention relates to a touch sensor device comprising: at least one one-dimensional sensor arranged to output a detection signal in response to the proximity of an object; a position processing unit for calculating the position of an interaction with the sensitive area from an analysis of the detection signals and for outputting a time sequence of data on the interaction positions on the sensor; and a gesture processing unit operable to analyze the temporal sequence of the data and to distinguish one or more gesture inputs, the gesture processing unit being encoded with a gesture recognition code comprising a plurality of linked state modules. The invention also relates to a corresponding signal processing method.

The gesture recognition code can be written in a high level programming language such as C and then compiled and embedded in a microcontroller chip or a CPU chip. The gesture recognition code is preferably loaded into the same chip that also interprets the touch signals from the screen and generates the time sequence of data, such as in a microcontroller or in another programmable logic device such as an FPGA (Field Programmable Gate Array). This approach has been used to generate reliable and testable code for single-touch data entry screens as well as multi-touch data entry screens. In the case of a single-touch screen, only simultaneous touching on the screen is supported, with only one x, y coordinate being output at a time. A multi-touch data entry screen can capture multiple simultaneous touches, such as two or three simultaneous touches.

The state machine comprises an idle state module, which corresponds to the start state and is also the state to be returned to after a gesture interpretation state module has been exited.

In response to a touch, the idle state gives control to a touch state.

In a multi-touch environment, the state machine in the second embodiment described below is implemented such that there are multiple touch states, namely one for a single touch, one for a double touch, one for a triple touch, etc., with control based on the number of simultaneous touches, which are defined by the time sequence of data at this point in time, is given to the corresponding touch state.

The approach described above for handling multiple-touch gestures through double-touch and triple-touch states associated with single-touch states works well, but reconfiguring the state machine to add a new multiple-touch gesture is difficult given the increasingly complex web of states and transitions. This problem is solved by a fourth embodiment of the invention, described further below, in which a plurality of state machines are provided, each of which is limited to single-touch gesture recognition. If the gesture recognition code is configured to recognize gestures with up to three simultaneous touches, then three such single touch state machines are provided. Further state machines are provided for multi-touch gesture recognition, each of which is responsible for a certain number of simultaneous touches, so that a double-touch state machine and optionally a triple-touch state machine and further optional additional state machines are provided for even larger numbers of simultaneous touches. A major advantage of this approach is that it uses the same code base for handling single touches, processing double touches, triple touches, or larger numbers of simultaneous touches using separate, dedicated code embedded in separate state machines.

A touch is usually output as a valid touch when certain criteria are met, with a common criterion being that there is a sequence of touches at a stable x, y position or in an x, y range over multiple time sample increments. If a touch with a duration of more than a threshold duration is detected in the touch state, then the control flow goes to a push state module, the push state being used to handle longer touches. The push state is preferably a superstate that includes multiple substates to distinguish between different durations of presses and / or allow a very long press to be interpreted as a repeated press, which can be useful, for example, for applications with input via alphanumeric keys.

The state machine preferably also has a plurality of state modules for interpreting high-level gestures, such as one or more states for interpreting double-tapping, short-dragging, dragging, and other gestures. The gestures usually include those specifically described in this document as well as other gestures known from the prior art, such as in particular those indicated in the prior art documents cited above.

The invention provides, in one aspect, a touch sensor device comprising: a sensor having a sensitive area extending in at least one dimension and arranged to output detection signals in response to the proximity of an object to a sensitive area; a position processing unit which can be operated to calculate positions of interactions with the sensitive area from an analysis of the detection signals and to output a time sequence of data indicating the interaction positions of the sensor and contacts therewith; and a gesture processing unit operable to analyze the temporal sequence of data to distinguish one or more gesture inputs therefrom, the gesture processing unit being encoded with a gesture recognition code comprising a plurality of linked state modules.

Further aspects of the invention relate to the gesture processing unit alone and the gesture processing unit in combination with the position processing unit, but without the sensor.

The plurality of state modules preferably comprises an idle state module and a plurality of gesture interpretation state modules, the idle state module being entered at the start of operation and being able to be returned to by at least some of the gesture interpretation state modules. The plurality of gesture interpretation state modules may include a touch state module for simple touches, wherein the post-touch idle state gives control to the touch state.

In some embodiments, the plurality of gesture interpretation status modules includes at least one multi-touch status module operable to process multiple simultaneous touches, the gesture processing unit operable to provide control based on the number of simultaneous touches made by the temporal sequence of data is defined at this point in time to be delivered to the corresponding touch status module. A multi-touch status module can be provided for two simultaneous touches and three simultaneous touches, with larger numbers of touches optionally also being able to be taken into account.

The plurality of gesture interpretation state modules may preferably include a push state module to which control can be given by a touch state module when a touch with a duration longer than a threshold duration is detected in the touch state module. The pressing state is preferably a superstate which comprises several sub-states in order to distinguish between different pressing durations.

In some embodiments, the plurality of gesture interpretation state modules includes a plurality of state modules operable to recognize movement-related gestures derived from one or more moving touches. In other embodiments, only static gestures such as a press, tap or double tap are taken into account.

A preferred implementation for interpreting multiple touch gestures according to the invention provides a gesture recognition code configured to recognize gestures with up to N simultaneous touches, where N is at least equal to 2, and comprises N single-touch state machines that can be operated, to recognize only single touch gestures, and N-1 multi-touch state machines, each of which can be operated to recognize only n-fold touch gestures, where n equals 2 to N.

The position processing unit and the gesture processing unit can be accommodated and implemented in a single integrated circuit such as a microcontroller. Alternatively, the position processing unit can be accommodated and executed in a first integrated circuit such as a microcontroller, while the gesture processing unit can be implemented in one or more separate integrated circuits such as a PC or another complex system with its own processing unit, a graphics processing unit and / or a digital one Signal processor with associated memory and bus connections can be accommodated and executed.

In another aspect, the invention provides a method for recognizing gestures from a temporal sequence of touch data with coordinates of interaction positions on a touch sensor, the method comprising: receiving touch coordinates that are temporally labeled or ordered; Analyzing the touch coordinates in a state machine comprising a plurality of associated state modules to recognize one of a plurality of defined gestures therefrom; and outputting the recognized gestures.

In a further aspect, the invention provides a single integrated circuit which has a memory onto which the above-mentioned gesture state machine is loaded and which can be operated to carry out the above-defined method for gesture recognition.

In a further aspect, the invention provides a computer which has a memory onto which the above-mentioned gesture state machine is loaded and which can be operated in order to carry out the above-defined method for gesture recognition.

It should be noted that the gesture state machine approach to gesture recognition can be applied to any hardware platform. Capacitive touch sensors, and particularly one-dimensional and two-dimensional capacitive touch sensors, are an important type of sensor that can provide a hardware platform for a gesture recognition state machine in accordance with the invention. In particular, the invention can also be applied to what are known as passive or active capacitive detection techniques.

Passive capacitive sensing devices measure the capacitance of a sensing electrode to a system reference potential (earth). The principles of this technique are for example in U.S. 5,730,165 and U.S. 6,466,036 described. In general, passive capacitive sensors use sensing electrodes connected to capacitive measurement circuits. Each capacitive measurement circuit measures the capacitance (capacitive coupling) of its associated sensing electrode to a system ground. When there is no pointing object in the vicinity of the sensing electrode, the measured capacitance has a background or quiescent value. This value depends on the geometry and layout of the sensing electrode and the connection lines to the same, etc., as well as the nature and position of neighboring objects such as the proximity of the sensing electrodes to neighboring ground planes. When a pointing object such as an operator's finger is brought near the sensing electrode, the pointing object appears as a virtual earth. This increases the measured capacitance of the sense electrode to earth. Such an increase in the measured capacitance indicates the presence of a pointing object. U.S. 5,730,165 and U.S. 5,466,036 are primarily designed for discrete measurements (with a single button) and not for two-dimensional applications of position sensors. In the U.S. 5,730,165 and U.S. 5,466,036 However, the principles described can easily be applied to 2D capacitive touch sensors (2DCTs), for example by providing electrodes to define a 2D array of discrete sensing areas or rows and columns of electrodes in a matrix configuration.

Active 2DCT sensors measure the capacitive coupling between two electrodes (rather than between a single sensing electrode and a system ground). The principles underlying the active capacitive detection techniques are discussed in U.S. 6,452,514 described. In an active sensor, an electrode, the so-called control electrode, is supplied with an oscillating control signal. The degree of capacitive coupling of the drive signal to the sense electrode is determined by measuring the amount of charge transferred to the sense electrode by the oscillating drive signal. The amount of charge transferred, ie the strength of the signal detected at the detection electrode, is a measure of the capacitive coupling between the electrodes. If there is no pointing object in the vicinity of the electrodes, the measured signal at the detection electrode has a background or resting value. However, when a pointing object, such as a user's finger, is brought near the electrodes (and specifically brought into the vicinity of the area between the electrodes), the pointing object acts as a virtual earth and draws part of the drive signal (the charge). from the drive electrode. This reduces the strength of the component of the drive signal coupled to the sense electrode. Such a reduction in the measured signal at the sensing electrode indicates the presence of a pointing object.

It should be noted that there are various other acquisition technologies as well. These include techniques based on resistive screens, usually operated with a pen, or technologies developed for large areas such as those based on ultrasound or other acoustic techniques, or techniques based on total internal reflection or others optical effects are based. All of these techniques can benefit from the present invention.

It should be noted that the term “state machine” is used in the present patent application. A state machine can also be indicated by the acronym FSM (Finite State Machine).

• 1 zeigt eine Gestenerkennungs-Zustandsmaschine gemäß einer ersten Ausführungsform.
• 2 zeigt eine Gestenerkennungs-Zustandsmaschine gemäß einer zweiten Ausführungsform.
• 3 zeigt eine Gestenerkennungs-Zustandsmaschine gemäß einer dritten Ausführungsform.
• 4 zeigt eine Gestenerkennungs-Zustandsmaschine gemäß einer vierten Ausführungsform zur Handhabung von N Berührungen.
• 5 zeigt die internen Zustände einer Zustandsmaschine für N Berührungen gemäß der vierten Ausführungsform.
• 6 ist ein Beispiel für die vierte Ausführungsform und zeigt die Erzeugung von Zweifachberührungs-Ereignissen unter Verwendung von Einfachberührungs-Zustandsmaschinen.
• 7 ist ein Beispiel für eine Dreifachberührungs-Gestenhandhabung gemäß der vierten Ausführungsform.
• 8 ist eine schematische Draufsicht, die Teile eines Elektrodenmusters für einen zweidimensionalen kapazitiven Berührungsbildschirm (2DCT) zeigt.
• 9 ist eine Draufsicht auf einen 2DCT von 8 und zeigt das Elektrodenmuster und eine erste Schicht von Verbindungen am Rand des Elektrodenmusterbereichs für die Verbindung mit den y-Elektroden.
• 10 ist eine Draufsicht auf die 2DCT von 9 und zeigt das Elektrodenmuster und eine zweite Schicht von Verbindungen am Rand des Elektrodenmusterbereichs für die Verbindung mit den x-Elektroden und auch für die Verbindung der externen Zuführleitungen mit den y-Elektrodenverbindungen von 9.
• 11 ist eine schematische Ansicht auf der Systemebene, die eine Ansteuerungs- und Datenerfassungsschaltung für den 2DCT von 8-10 zeigt.
• 12 zeigt schematisch einen Anzeigemonitor und eine Eingabeeinrichtung gemäß der vorliegenden Erfindung.
• 13 zeigt schematisch ein Mobiltelefon gemäß der vorliegenden Erfindung.

The invention is illustrated in the following on the basis of some embodiments with reference to the accompanying drawings.

• 1 Figure 3 shows a gesture recognition state machine according to a first embodiment.
• 2 Figure 3 shows a gesture recognition state machine according to a second embodiment.
• 3 Figure 3 shows a gesture recognition state machine according to a third embodiment.
• 4th shows a gesture recognition state machine according to a fourth embodiment for handling N touches.
• 5 Fig. 13 shows the internal states of an N touch state machine according to the fourth embodiment.
• 6th Figure 13 is an example of the fourth embodiment and shows the generation of double-touch events using single-touch state machines.
• 7th Fig. 13 is an example of triple-touch gesture handling according to the fourth embodiment.
• 8th Fig. 13 is a schematic plan view showing parts of an electrode pattern for a two-dimensional capacitive touch screen (2DCT).
• 9 FIG. 13 is a top plan view of a 2DCT of FIG 8th and shows the electrode pattern and a first layer of connections at the edge of the electrode pattern area for connection to the y-electrodes.
• 10 FIG. 3 is a top plan view of the 2DCT of FIG 9 and shows the electrode pattern and a second layer of connections at the edge of the electrode pattern area for connection to the x-electrodes and also for connection of the external feed lines to the y-electrode connections of FIG 9 .
• 11 FIG. 13 is a system level schematic view illustrating drive and data acquisition circuitry for the 2DCT of FIG 8th - 10 shows.
• 12th Fig. 3 schematically shows a display monitor and an input device according to the present invention.
• 13th Figure 3 shows schematically a mobile phone according to the present invention.

Before various embodiments of the invention are described below, the gestures mentioned in the detailed description of the embodiments should first be explained here.

Tap: When tapping, a user quickly touches and leaves the touch area. No substantial movement occurs while the user's finger is on the touch surface. Typing is characterized by a short touch time. A tap can be used, for example, to activate a hyperlink on a displayed web page.

Double tap: With a double tap, the user touches and leaves the touch area twice in quick succession. No substantial movement occurs while the user's finger is on the touch surface. Double tapping is characterized by short touch times and a short pause between the first and second touch. For example, double-tapping can be used to highlight a word in a displayed document.

Press: When pressed, the user touches and holds the touch surface. No substantial movement occurs while the user's finger is on the touch surface. For example, the press can be used to select a number from a displayed keypad. The same mechanism can also be used to automatically repeat the dialed number if the user presses and holds the displayed number.

Short pull (flick): With a short pull, the user touches the touch surface quickly, makes a short movement and then lets go of the touch. The short pull is characterized by a short duration of contact. For example, a quick drag can be used to view the next in a sequence of images.

Drag: When dragging, the user touches the touch area, moves his finger over the area and ends the touch. Dragging is characterized by a large movement across the touch area. Depending on the application, multiple drag events can be generated while the user is holding his finger emotional. This can be used, for example, to mark a sentence in a displayed document.

Compression: In the case of compression, the user places two fingers on the touch surface and then moves the fingers towards each other. This can be used, for example, to reduce the size of a displayed image.

Stretch: In stretching, the user places two fingers on the touch surface and then moves the fingers apart. This can be used, for example, to enlarge a displayed image.

Rotate: When rotating, the user places two fingers on the touch surface and then rotates the fingers relative to each other. This can be used, for example, to rotate a displayed image.

Before various embodiments of the invention are described below, the events mentioned in the detailed description of the embodiments should first be defined here. These events are carried out by the state machines of the invention described below. It should be noted that, in some embodiments, the state machine can simultaneously generate multiple events in response to certain user actions.

Tap: A tap event is generated when the user makes a tap gesture.

Double tap: A double tap event is generated when the user makes a double tap gesture.

Press: A press event is generated when the user touches the touch area and then holds their finger stationary.

Short press: A short press event is generated after a press event if the user continues to hold their finger stationary for a certain period of time.

Long press: A long press event is generated after a short press event if the user continues to hold their finger stationary for a certain period of time.

Repeated Press: A repeated press event is generated after a long press event when the user has held their finger stationary for a certain period of time. If the user continues to hold his finger stationary, repeated press events will be generated at regular intervals.

Release: A release event is generated when the user removes their fingers from the touch area.

Short Drag (Flick): A short drag event is generated when the user makes a short drag gesture.

Drag: A drag event is generated when the user makes a drag gesture.

Multi-touch (or double-touch): A multi-touch (or double-touch) event is generated when the user touches the touch area with two fingers.

Compress: A compress event is generated when the user performs a compress gesture.

Stretch: A stretch event is generated when the user makes a stretch gesture.

Rotate: A rotation event is generated when the user performs a rotation gesture.

1 Figure 12 shows a state machine for gesture recognition according to a first embodiment that is limited to processing gestures consisting of simple touches. In other words, multi-touch gestures are not taken into account in this embodiment. The supported gestures are: tapping; To press; double tap; short pull (flick); and pulling. These five gestures are recognized in the state machine by going through seven states which, as shown in the figure, are connected to one another by approximately 17 transitions. The states can be further divided into those for static touches such as pressing, tapping and double tapping in this example, and those for moving touches such as brief dragging and dragging in this example.

The following describes the idle state. The idle state always occurs when the touch surface is not touched by a user. In this state, no actions are triggered and no actions are carried out. The idle state changes to a touch state when the user touches the touch surface with a finger. The idle state has no parameters and no timing calculations are associated with the idle state.

The state of contact is described below. The touched state occurs when a User touches the touch surface. When the touch state is entered, a time restriction is started, ie a timer is set which expires after a predetermined period of time. This distinguishes between short-duration gestures such as a short flick and tapping, and long-duration gestures such as pressing. The starting position of the touch is saved. This is used to determine whether the user has moved their fingers. In the touch state, it is checked whether the time limit has expired and the distance traveled from the initial position of the touch is calculated. When the time limit expires, the state machine transitions by generating a push event and then going into the push state. When the user moves the finger a specified distance from the home position of the touch, the state machine enters a wait state for a short drag. When the user finishes touching, the state machine enters the wait state for the second tap. The following actions determine whether a tap, a double tap or another gesture is generated. The time constraint parameter in the touch state determines how long the user must remove their finger in order to generate a tap event. The movement parameter in this state determines the distance over which the user must move his finger in order to enter the wait state for a short drag. The maximum allowable amount of time from the first touch for an event to be generated in this state is the time limit period.

The waiting state for the second tap is described below. The waiting state for the second tap occurs after a user taps the touch area. A time limit is started when the wait state is entered. This is used to determine whether the user has made a tap gesture. The starting position of the touch is saved. This is to determine whether the user is making a double tap gesture. The expiry of the time limit is checked. When the time limit expires, a tap event is generated and the state machine enters the idle state. This means that the user tapped the surface once, but not twice, during the allowed time interval. When the user presses the touch area with a finger, the distance of the second touch from the first touch is calculated. If the distance is less than the specified value, the state machine enters the double tap wait state because the user may be double tapping. If the distance between the first touch and the second touch is greater than the specified value, the state machine generates a tap event (for the previous tap gesture) and re-enters the touch state as shown. The time constraint parameter in the wait state for the second tap determines how long after a touch by the user is finished, a tap event is generated. If the user touches the surface again before the time limit expires, a double tap event can be generated. The maximum allowable time from the first touch to the generation of an event in this state corresponds to the sum of the time restriction periods of the touch state and the waiting state for a second tap.

The waiting state for a double tap is described below. The double tap wait state occurs when the user has tapped the touch area and then presses it again. When entering this state, a time limit is started. This is to determine if the user made a double tap gesture. The starting position of the touch is saved. This is used to determine whether the user has moved their finger. In this state, it is checked whether the time limit has expired and the distance covered from the starting position of the touch is calculated. When the time limit expires, tap and push events are generated and the state machine enters the push state. If the user moves more than a specified distance from the initial position of the touch, the state machine generates a tap event and enters the wait state for a short flick. The reason for this is that the user has typed once and a possible second tap is now a possible short flick. When the user ends the touch, the state machine generates a double tap event and enters the idle state. This is because the user performed both taps within the specified time and distance limits. The timeout parameter in this state determines how long the user must remove their finger in order to generate a double tap event. When the time limit expires, it is assumed that the user typed and then pressed and held the surface. The movement parameter in this state determines how far the user has to move his finger in order to enter the wait state for a short drag. The maximum allowable time from the first touch for the generation of an event in this state corresponds to the sum of the time restrictions of the touch state, the wait state for a second tap and the wait state for a double tap.

In the following the Waiting state for a quick pull (flick) described. The quick flick wait state occurs when the user has pressed the touch pad and moved their finger across the surface. When this state is entered, a time limit is started. This is used to determine whether the user is making a quick drag gesture (flick) or a drag gesture. When entering, the starting position of the touch is also saved. This is forwarded to the processing of waiting for a pull, if necessary. In this state, a check is made to see if the time limit has expired. When the time limit expires, a push event is generated and the state machine enters the wait state for a pull. This is because the user did not finish touching within the time used to distinguish between a quick flick and a drag. When the user ends the touch, the state machine generates a quick flick event and enters the idle state. This is because the user completed the move within the specified time limit. The time constraint parameter in this state determines how long the user must remove their finger in order to generate a quick flick event. When the time limit expires, it is assumed that the user is performing a drag operation. The maximum length of time from the first touch to generating an event in this state is the sum of the time limit periods of the touch state and the wait states for a quick flick.

The following describes the waiting state for dragging. The drag wait state occurs when the user has pressed the touch pad and moved their finger across the surface, the motion taking longer than the time for a quick flick. When this state is entered, a time limit is started. This is used to determine whether the user has held their finger stationary. The starting position of the touch is also saved upon entry. This is used to determine whether the user has moved their finger. In this state, it is checked whether the time limit has expired and the distance from the starting position of the contact is calculated. When the time limit expires, the state machine enters the push state. This is because the user kept his finger stationary for too long. If the user moves their finger more than a specified distance from the home position of the touch, the state machine generates a drag event. When the user ends the touch, the state machine generates a termination event and enters the idle state. The time constraint parameter in this state determines how long the user's finger must remain stationary before entering the push state. The movement parameter in this state determines how far the user has to move his finger in order to generate a drag event. The maximum allowable time from the first touch for the generation of an event in this state corresponds to the sum of the time restrictions of the touch state, the wait state for a quick flick and the wait state for a drag.

The push status is described below. The press condition occurs when a user has pressed the touch pad and held their finger stationary for a certain amount of time. When entering this state, the initial position of the contact is saved. This is used to determine whether the user has moved their finger. In this state, the distance from the starting position of the contact is calculated. If the user has moved more than a specified distance from the home position of the touch, a drag event is generated and the state machine enters the wait state for a drag. When the user releases the touch, the state machine generates a release event and enters the idle state. The movement parameter in this state determines how far the user must move his finger in order to generate a drag event and enter the wait state for a drag. There is no limit to how long it can take before an event is generated in this state.

It should be noted that a simplified version of the first embodiment can be provided in which moving gestures such as flick and drag are not supported.

2 13 shows a state machine for gesture recognition according to a second embodiment, which can be viewed as a development of the first embodiment with an extension for multi-touch gestures. The same five single-touch gestures are supported as in the first embodiment, plus three multiple-touch gestures, namely compressing, rotating and stretching. The multi-touch capability is provided by an additional state, the multi-touch state, and approximately eleven additional transitions as shown in the figure. This embodiment will be described below with particular attention to the modifications related to the first embodiment.

First, the additional state of multiple touch (double touch) will be described. The multi-touch condition occurs when the user has placed two fingers on the touch surface. Upon entry, the initial positions of the touches and the angle between the initial positions of the touches are saved. In this state, the distance between the two simultaneous touches and the angle between the touch positions are calculated. When the user releases both fingers, the state machine transitions by generating a resolution event and entering the idle state. When the user releases one or generally all but one finger, the state machine enters the press state. If the touch distance increases by more than a specified distance from the first touch distance, a stretch event is generated. If the contact distance is decreased by more than a specified distance from the first contact state, a forge event is generated. If the angle between the touch positions changes by more than a specified amount from the original angle, a rotation event is generated. It should be noted that in this state the state machine can also generate several events, for example stretching and rotating or compressing and rotating. The angle parameter in this state determines how much the angle must change between the touch positions in order to create a rotation event. The stretching threshold parameter in this state determines how much the distance between the two touch positions must increase in order to generate a stretching event. The compression threshold parameter in this state determines how much the distance between the two contact positions must decrease in order to generate a compression event. There is no limit to how long it can take before an event is generated in this state.

An additional transition to the multiple touch state is provided from the idle state. This occurs when the user presses the touch surface with two fingers, in which case the state machine generates a multi-touch event and enters the multi-touch state.

An additional transition from the touch state to the multiple touch state is provided. This occurs when the user places a second finger on the touch area, in which case the state machine generates a press event and a multiple touch event and enters the multiple touch state.

An additional transition to the multiple touch state is provided from the waiting state for a second tap. This occurs when the user touches the touch surface with two fingers, in which case the state machine generates a tap event and a multiple touch event and enters the multiple touch state. This indicates that a single tap was followed by a multiple touch.

An additional transition to the multiple touch state is provided from the waiting state for a double tap. This occurs when the user touches the touch surface with two fingers, in which case the state machine generates a tap event, a press event and a multiple touch event and enters the multiple touch state. This indicates that a single tap, a press, and then a multiple touch followed each other.

From the waiting state for a quick flick, an additional transition to the multiple touch state is provided. This occurs when the user presses the touchpad with two fingers, in which case the state machine generates a quick drag event and a multi-touch event and enters the multi-touch state. It is assumed that the multiple touch has ended the quick drag gesture.

An additional transition to the multiple touch state is provided from the waiting state for a pull. This occurs when the user presses the touch surface with two fingers, with the state machine generating a multi-touch event and entering the multi-touch state.

An additional transition is provided from the pressed state. This occurs when the user presses the touch surface with two or more fingers, with the state machine generating a multi-touch event and entering the multi-touch state. If the user continues to release everything but one finger in the multi-touch state as mentioned above, the state machine enters the press state.

3 FIG. 13 shows features of a state machine for gesture recognition according to a third embodiment, which can be regarded as a development of the second embodiment in which the pressing state has been further developed. In this embodiment, the push state is modified to a super state that includes five states. The additional states are a short wait, a long wait, a wait for a retry, a retry, and a completed press. These additional states allow multiple interpretations of a touch depending on its duration.

The short wait state is an initial state of the super state which is entered by the other states of the state machine when a user has pressed the touch area and held his finger stationary for a certain period of time. Upon entering the short wait state, a time limit is started that is used to determine when a short press event should be generated. The initial position of the touch is saved and used to determine if the user moved their finger. In the short wait state, it is checked whether the time limit has expired and the moving distance from the starting position of the touch is calculated. When the time limit expires, a short press event is generated and the state machine enters the long wait, wait for a retry, retry, or complete press. If the user moves the finger more than a specified distance from the initial position of the touch, a drag event is generated and the state machine enters the wait state for a drag. When the user ends the touch, the state machine generates a termination event and enters the idle state. When the user presses the touchpad with two fingers, the state machine generates a multi-touch event and enters the multi-touch state. The time constraint parameter in this state determines how long the user's finger must remain stationary before moving to the long wait state, the wait for a retry state, or the completed press state. The movement parameter in this state determines how far the user must move his finger in order to generate a drag event and enter the wait state for a drag. The maximum allowable time from the first touch for the generation of an event in this state corresponds to the sum of the time restrictions of the touch state, the wait state for a quick pull (flick), the wait state for a pull and the short wait state.

The long wait state is provided for when a user has pressed the touch surface and kept his finger stationary for a certain time. A time limit is started upon entry. This is used to determine when a long press event should be generated. With the entry, the starting position of the contact is also saved. This is used to determine whether the user has moved their finger. In this state, it is checked whether the time limit has expired, the distance from the starting position of the contact being calculated. When the time limit expires, a long press event is generated and the state machine enters the wait state for a retry or the complete press state. If the user moves their finger more than a specified distance from the initial position of the touch, a drag event is generated and the state machine enters the wait state for a drag. When the user ends the touch, the state machine generates a termination event and enters the idle state. When the user presses the touchpad with two fingers, the state machine generates a multi-touch event and enters the multi-touch state. The time constraint parameter in this state determines how long the user's finger must remain stationary before entering the wait state for a retry or the complete wait state. The movement parameter in this state determines how far the User must move his finger to generate a drag event and enter the wait state for drag. The maximum time from the first touch for the generation of an event in this state corresponds to the sum of the time restrictions of the touch state, the wait state for a quick flick, the wait state for a drag, the short wait state and the long wait state.

The waiting state for a repetition is provided that a user has pressed the touch surface and kept his finger stationary for a certain period of time. A time limit is started when the state is entered. This is used to determine when to generate a repeated push event. With the entry, the starting position of the contact is also saved. This is used to determine whether the user has moved their finger. In this state, it is checked whether the time limit has expired and the distance from the starting position of the contact is calculated. When the time limit expires, a repeat press event is generated and the state machine enters the repeat state. If the user moves their finger more than a specified distance from the initial position of the touch, a drag event is generated and the state machine enters the wait state for a drag. When the user ends the touch, the state machine generates a termination event and enters the idle state. When the user presses the touchpad with two fingers, the state machine generates a multi-touch event and enters the multi-touch state. The time constraint parameter in this state determines how long the user's finger must remain stationary before entering the repeat state. The movement parameter in this state determines how far the user must move his finger in order to generate a drag event and enter the wait state for a drag. The maximum time from the first touch for the generation of an event in this state corresponds to the sum of the time restrictions of the touch state, the wait state for a short pull (flick), the wait state for a pull, the short wait state, the long wait state and the wait state for a repetition.

The repetition state is provided that the user has pressed the touch surface and kept his finger stationary for a certain period of time. A time limit is started upon entry. This is used to determine when to generate a repeated push event. With the entry, the starting position of the contact is also saved. This is used to determine whether the user has moved their finger. In this state, the expiration of the time limit is checked and the moving distance from the starting position of the touch is calculated. When the time limit expires, a repeated push event is generated. If the user moves their finger more than a specified distance from the initial position of the touch, a drag event is generated and the state machine enters the wait state for a drag. When the user ends the touch, the state machine generates a resolution event and enters the idle state. When the user presses the touchpad with two fingers, the state machine generates a multi-touch event and enters the multi-touch state. The time constraint parameter in this state determines how long the user's finger must remain stationary before a repeated press event is generated. The movement parameter in this state determines how far the user must move his finger in order to generate a drag event and enter the wait state for a drag. The maximum time from the first touch for the generation of an event in this state corresponds to the sum of the time restrictions of the touch state, the wait state for a quick pull (flick), the wait state for a pull, the short wait state, the long wait state and the recovery state .

The completed pressing state is similar to the pressing state of the first and second embodiments. With the entry, the starting position of the contact is saved. This is used to determine whether the user has moved their finger. In this state, the moving distance from the initial position of touch is calculated. If the user has moved their finger more than a specified distance from the initial position of the touch, a drag event is generated and the state machine enters the wait state for a drag. When the user ends the touch, the state machine generates a resolution event and enters the idle state. When the user presses the touchpad with two fingers, the state machine generates a multi-touch event and enters the multi-touch state. The movement parameter in this state determines how far the user must move his finger in order to generate a drag event and enter the wait state for a drag. There is no limit to how long it can take before an event is generated in this state.

It should be noted that not all in 3 The transitions shown in relation to the superpressure state are active at the same time. Each of the push sub-states can be deactivated if necessary. Then the internal paths through which the substates are activated are determined. For example, if a short press and a repeat press are enabled but a long press is disabled, then a path in the superstate is to the brief wait state. The long wait state is skipped in this case, and the machine goes to the wait state for one retry.

It should be noted that the first embodiment can also be modified to have a super-press state as above with reference to FIG 3 is provided, but the features for a multiple touch are omitted.

The state machines of the first, second and third embodiments are simple and effective, but do not scale well for more complex combinations of gestures, especially when it comes to multiple touches. For example, consider a gesture in which a user holds one finger on the touch surface and taps the surface with a second finger (a “press” and “tap” gesture). For this purpose, the multiple touch state in the state machine of the second embodiment requires multiple sub-states in order to track the position, the duration and the movement of the two touches.

It would therefore also be difficult to extend the state machine of the second embodiment for the handling of gestures with more than two touches. The complexity of the state machine would greatly increase with each additional touch.

These limitations are eliminated by a fourth embodiment of the invention, in which the multi-touch state from the State machine of the second embodiment is removed to provide a simple single touch state machine according to the first embodiment as shown in FIG. A single touch state machine of this type can keep track of all required information from a single touch. A plurality of single-touch state machines are then combined with one another in order to handle multiple-touch gestures, so that there is no need to generate more complex, multiple-touch state machines that follow the approach of the second embodiment.

4th Fig. 13 shows a state machine according to the fourth embodiment at the highest conceptual level. In this diagram, the x touch blocks each give the single-touch state machine of 3 again. N such state machines are processed by an N multiple touch state machine to generate N multiple touch gestures.

5 Fig. 13 shows the internal structure of the multiple-touch state machine of the second embodiment, which includes an idle state and a touch state. If all the touches are made, the state machine is in the multiple-touch idle state, otherwise it is in the multiple-touch idle state. The state machine can generate events in any state. For example, in the multiple touch touch state, if all of the single touch state machines are in the wait state for a drag, a multiple touch drag event may be generated when one of the state machines generates a drag event.

6th shows how this approach can be used to provide an equivalent function for the state machine of the second embodiment of FIG 2 but to be provided with a simpler and scalable architecture.

In this example, touch 1 and touch 2 are processed by the two touch state machine, which tracks the distance and angle between touches and creates stretch, compress, and twist events as the distance and / or angle between touches changes changes.

This state machine can also generate more complex gestures such as the previously described press and tap gestures. This can be done using a combination of the states and generated events of the two input state machines. If one state machine is in the "press" state and the other has generated the event "tap", then the two-touch state machine can generate a "press and tap event".

• •Drücken und doppeltes Tippen
• •Ziehen und Tippen
• •Drücken und schnelles Ziehen (Flick)
• •Ziehen mit zwei Fingern
• • andere Kombinationen von Zuständen und Ereignissen.

Similar combinations can be used to generate events such as the following:

• • Press and double tap
• • Drag and tap
• • Push and pull quickly (flick)
• • Dragging with two fingers
• • other combinations of states and events.

7th shows how the same approach can be used to handle three-touch gestures. The figure shows three possible triple touch gestures. In the triple-touch state, if all of the single-touch state machines are waiting for a drag, then the triple-touch state machine may generate a triple-touch drag event when any single-touch state machine generates a drag event.

The figure also shows how the state machine can detect a situation in which the user has two touches on the touch surface, in which case he then taps with a third finger. This can be used, for example, in an application where two fingers are used to rotate a displayed image and the displayed image is enlarged by tapping a third finger. The same state machine can also be used to simply capture a double tap to zoom out on the displayed image.

A main advantage of the approach of the fourth embodiment is that the same code base is used for the handling of single touches, with double touches, triple touches or even larger multiple touches being processed using a separate and additional code. The approach is therefore scalable and modular and thus brings the usual advantages in terms of reduced programming effort and greater reliability.

A suitable hardware platform for integrating the gesture recognition state machine according to the invention is described below. The hardware platform is from the one mentioned above US 2007/0279395 A1 accepted.

8th Fig. 13 is a schematic plan view of important parts of an electrode pattern for a 2DCT, the electrode pattern being a sensitive area defined for the facility. The electrodes are arranged on a substrate, not specifically shown, which has a surface which lies in the plane of the drawing. The substrate can be made from a flexible and transparent plastic material such as PET (polyethylene terephthalate). The substrate is generally insulating. The electrode pattern is made of indium tin oxide and has a resistance of a few hundred ohms / square. This is a transparent material, so it is suitable for display applications or other applications in which an underlying button or other template should be visible.

In general, the electrode pattern can be made by applying or removing a suitable conductive material. The application can take place, for example, by means of gas phase deposition or screen printing. The removal can be carried out, for example, by a laser or by chemical etching.

The electrode pattern defines y-electrodes 10, 12 for determining the y-position and the x-electrodes 14, 16, 18, 20, 22, 24 for determining the x-position. As shown, the x-electrodes and the y-electrodes extend generally in the x-direction and are interlaced with one another in the y-direction. The y-electrodes 10, 12 are designed as simple bars, i.e. H. as elongated rectangles, while the x electrodes 14-24 each have tapered triangular shapes.

In the following, the x-electrodes are first described in detail. This is followed by a description of the y-electrodes.

The x-electrodes can be classified into three groups. A first group of triangular, tapered electrodes 14th , 24 is located on the left and right of the sensitive area. A second group of triangular electrodes 16 , 22nd having a double-tapered shape is arranged such that the electrodes extend inwardly toward the center from each of the left and right sides of the sensitive area.

A third group of x-electrodes 18, 20 each extend from a center line 26th left and right to the outside and are integrally formed. Adjacent elements of the first and second group 14th , 16 and 24 , 22nd extend together in the x-direction over the outer parts I and IV of the sensitive area to the left and right side of the sensitive area. And neighboring elements of the second and third groups 16 , 18th and 22nd , 20th extend in the x-direction over inner parts II and III of the sensitive area on either side of the center line.

In this way, each adjacent pair of mutually extending x-electrodes of the first and the second group or the second and the third group forms a so-called slide. It should be noted that the electrode elements are appropriately shaped and dimensioned to correspond to the actuating device such as a finger in order to provide a ratiometric, capacitive signal over the length of the mutual extension in the x-direction so that they are mutually in the Overlap x-direction.

The left, double-tapered x-electrodes 16 are all over an electrical line 30th extending in the y-direction on the left edge of the sensitive area near the left edge of the x electrodes 16, with an external lead X1 tied together. It should be noted that the double-tapered electrodes have connection areas at their left-hand ends 33 to support this external connection.

The left tapered x-electrodes 14 are all over an electrical line 32 running in the y-direction on the left edge of the sensitive area near the left edge of the x electrodes 14, with an external lead X2 tied together.

The tapered x electrodes 18 and 20 extending from the center line 26th extend, of course, are all connected to the center line and are across the center line 26th in electrical contact with the edge of the sensitive area. An external line X3 is via an electrical line 34 that is with the base of the center line 26th is in contact, connected to the center line.

The right tapered x electrodes 24 are all via an electrical lead 36 running in the y-direction on the right edge of the sensitive area near the right edge of the x electrodes 24, with an external lead X4 connected, similar to the corresponding left x electrodes 14.

The right double-tapered x-electrodes 22 are all over an electrical line 38 running in the y-direction on the right edge of the sensitive area near the right edge of the x electrodes 22, with an external lead X5 connected, similar to the corresponding left x-electrodes 16, with enlarged contact areas 39 be used.

In this way, the x electrodes 14-24 are externally connected to five external leads X1-X5 connected for readout.

The y electrodes are in two sets 10 and 12th left and right of the center line 26th divided up. As mentioned above, they have a simple bar shape and are arranged between adjacent sets of x-electrodes 14, 16, 18 on the left and 20, 22, 24 on the right. The y-electrodes 10 and 12 are connected to one another in vertically adjacent sets by electrical lines, so that the y-resolution of the sensitive area is limited to a vertical distance that corresponds to the vertical extension of the connected y-electrodes. Chaining the y-electrodes reduces the y-resolution, but the number of external lines for the y-electrodes is kept small. In the example shown, the bottom set of y-electrodes, which consists of four pairs of y-electrodes, is with an electrically conductive rail 50 connected that is part of an external line Y1 is. Although it cannot be seen in the figure, each pair of y-electrodes is jointly connected to an external circumferential rail at the same level. The next set consists of three pairs of y-electrodes, but only the first pair is shown, and is with a rail 52 for another external line Y2 tied together. There are a total of seven sets of y-electrodes with their corresponding external leads Y1-Y7 connected by associated conductive rails. The y-value is taken from these seven external lines, which only provide seven units of y-resolution for a simple control algorithm, but greater y-resolution can be achieved with interpolation between adjacent y-lines.

In summary, it can be said that the 2DCT provides a quasi-continuous x-resolution via sliders in four overlapping areas I-IV in the x-direction over the width of the sensitive area in combination with a stepwise y-resolution via horizontally extending electrode rods, which are connected together in vertically adjacent sets of 3 and 4. A total of twelve external lines are used, five for X and seven for Y.

The combination of the center line and the double-tapered electrodes allows a large area of the sensitive area in the x-direction to provide a large sensitive area that can be provided transparently and without external connections other than the edge. Furthermore, with this electrode structure, the hand shadow effects are not noticeable because a shift of the center of the capacitive signal from the physical position of the finger is limited by the lateral extension of the electrodes. For example, a 6 inch (150 mm) diagonal sensitive area device can be made based on this structure.

9 Figure 12 is a plan view at approximately 1: 1 scale (ie, full size) of a 2DCT with the electrode pattern and a first layer of connections at the edge of the electrode pattern area for connection to the y-electrodes. For the sake of simplicity, the area shown by the preceding schematic view is shown at the bottom of the figure by the rectangle surrounded by a double-dotted dashed line. A finger outline, roughly to scale, is also shown.

The general ITO electrode pattern covers most of the substrate 40 . The pattern in this example covers a rectangular area corresponding to the area of the touch screen or other device for which a sensing part is to be provided. The four overlapping areas I-IV of the x electrodes described above are also marked. The substrate 40 , which is generally rectangular, also has a protruding tab 42 in the middle of the left side of the substrate. The protruding tab 42 is used for external contacting as described with reference to the following figure. On the left side of the substrate 40 , ie on the side next to the protruding flap 42 , are seven groups of senior rails 50-62 provided the external lines Y1-Y7 for the y-lines, with the lines Y2-Y7 each over the rails 52-62 connected to three y-electrodes and the lead Y1 four y-electrodes over the rail 50 connects so that a total of 22 y-electrodes are on the left half of the device, ie in the half to the left of the center line 26th , are provided. On the right-hand side there is an exactly corresponding arrangement with 22 y-electrodes, each in groups of three and a bottom group of four. The rails 50-62 of the external lines Y1-Y7 on the right side of the substrate extend around the top of the substrate to the left side of the substrate so that the corresponding pairs on the left and right sides and the commonly connected groups of pairs of y-electrodes are interconnected via a single conductive bar are connected.

10 FIG. 6 is a top plan view of the 2DCT prototype of FIG 9 and shows the electrode pattern and a second layer of connections at the edge of the electrode pattern area for connecting the x-electrodes and also for connecting the external leads of the y-electrodes to the y-electrode connections of FIG 9 . Between the first and second layers of 9 and 10 an insulation layer is disposed which provides insulating areas to prevent electrical contact between certain pairs of first and second interconnect layers and further provides open areas to allow electrical contact between certain other pairs of ensure first and second tie layers.

In the following, the y-electrodes are described first. Seven conductive rails 44 extend parallel in the x-direction along the upper part of the protruding tab 42 to the left of the main area of the substrate 40 . These then fan out and end in an enlarged contact area 46 directly over part of the rails 50-62 in the first link layer of 9 for each of the y-electrode connections Y1-Y7, allowing signals to and from each of the y-electrode groups via the external contact rails 44 can be input and output. There are open areas in the insulation layer at each contact field 46 provided to make electrical contact between each of the Y1-Y7 rails 44 on the second interconnect layer and the Y1-Y7 rails 50-62 to be provided in the first connection layer. In addition, insulating areas are provided in the insulating layer, which cover the Y1-Y7 rails where they lie over the ITO pattern, in order to contact the y-electrodes on the left and on the right side of the substrate.

The following describes the x-electrode connections. The five electrical lines 30-38 for the external connections X1-X5 have already been referring to 8th and are in the second connection layer of the prototype of 11 shown. As shown, the x-electrode connections are entirely provided on the second interconnection layer, while the y-electrode connections are distributed between the first and second interconnection layers. The rails 30-38 run around the bottom of the substrate 40 and then converge in five parallel rails that lead to the protruding tab 42 where they connect to the seven parallel y-electrode connections. It should be noted that the x-electrode connection bars and pads that run vertically up each side of the ITO area to contact the x-electrodes are electrically isolated from the y-electrode connection bars by the insulating layer.

11 Figure 13 is a system level schematic view of a multi-channel sensor circuit 140 for use with the touch screen. In this figure is the sensor circuit 140 with five capacitive electrode inputs X1 , X2 , X3 , X4 and X5 of the x-electrodes and a single capacitive electrode input Y _n representative of the seven y-electrode inputs. In reality, seven such leads are provided, one for each y-electrode input, to provide the total of twelve leads. A charge control line 157 is used to simultaneously all capacitive inputs X1-X5 and Y1-Y7 over with a reference voltage rail 158 connected charging switch 156 to charge.

The channels X1-X5 and Y1-Y7 work together to transfer charge to the charge detectors as shown in the figure by a single charge control line 163 used to switch the discharge 162 to operate to discharge all charged electrodes. After a transmission or a series of transmissions, an analog multiplexer selects 182 which of the charge detector capacitor outputs to the amplifier 184 and the analog / digital converter 186 is to be given.

It also uses an array of reset switches 188 through a reset control line 190 are activated after a pulse or a sequence of pulses in order to set the capacitive inputs to a known reference (e.g. to earth). The x channels X1-X5 must be operated and the signals must be processed in order to take into account the ratiometric information to be obtained from these signals.

Data processing then takes place on the digitized charge detector capacitor outputs. These outputs are stored in a processing unit 188 which includes a microprocessor and optionally a microcontroller for lower level processing.

In a first processing stage / unit 190, referred to as the touch coordinate determination stage / unit, the digitized charge detector capacitor outputs are collected and recorded as none, one or more simultaneous touches at defined x, y positions on the 2DCT for each repeated acquisition , interpreted each frame or each sampling interval. The touch coordinate determination stage / unit 190 has an output which can be operated to the analog multiplexer 182 and to cycle through the charge detector capacitor outputs, thereby obtaining the raw data from each of the x, y sensor node combinations. These data, ie the contact coordinates of each sample, are referred to as contact data sets.

In a second processing stage / unit 192, referred to as the touch tracking stage / unit, the touch records for successive samples are analyzed so that inputs resulting from a single touch moving from sample to sample are identified as such and new touches and past contacts can be distinguished. The data output from the touch tracking stage / unit are streams of x, y coordinates accumulated over time and determined over time tracked touches. These can be saved and / or reproduced in a tabular form.

When this data is input to a third processing stage / unit 194, referred to as a gesture recognition stage / unit, gesture recognition processing is carried out using the state machine code of the invention. The data output from the gesture recognition stage / unit are recognized gestures (including simple touches) and the times of their appearance.

In a fourth processing stage / unit 196, the data on the recognized gestures from the third stage are used in a code of a higher level, usually on the application level, but possibly also on lower levels of the coding hierarchy. These gestures are used to trigger a higher level function, such as the facility's graphical user interface and jumping between blocks of code in an application.

Further suitable capacitive hardware platforms for the detection of touch are for example in US 2005/0041018 A1 , US 2007/0247443 A1 and US 2007/0257894 A1 which give examples of passive capacitive sensors. For the sake of brevity, only one example is given in this document.

12th shows schematically a display monitor 901 and an input device 902 of a PC. In this example the input device comprises 902 an alphanumeric keyboard 903 through which the user enters characters into the PC, and a touchpad 904 . It should be noted that the touchpad 904 one separate from the keyboard 903 intended device can be. A conventional computer touchpad can also have a number of "mouse buttons" to enable the user to manipulate a cursor and select items by clicking one of the mouse buttons. Although you can move a cursor on the display with a conventional touchpad by moving a finger over the surface of the touchpad, and you can also operate a mouse button by tapping the surface of the touchpad with your finger, the conventional touchpad can do both of these Do not perform functions at the same time. It should be noted that the PC is also in the display monitor 901 can be included and that the touchpad 904 can also be one of the input devices of a laptop computer.

The touchpad 904 allows the user to use two separate fingers to perform two different functions on the PC. For example, the user can put one on the display monitor 901 Move the displayed cursor using one finger on the surface of the touchpad, while he can use another finger to move on the surface of the touchpad 901 similar to tapping a mouse button to highlight items. This can be used, for example, to organize files, starting with the one on the display monitor 901 The cursor displayed is moved with a first finger on the surface of the touchpad until it is positioned over the desired file. The file is then marked by placing a second finger on the surface of the touchpad. The file is then moved to a different location by moving the first finger across the surface of the touchpad while keeping the second finger in contact with the surface of the touchpad. When the file has reached the desired new position, the second finger is released from the surface of the touchpad, which removes the marking of the file. It should be noted that in the description above, the second finger can also be moved to the first finger at the same time, so that the user can perform this function with one hand.

Another function that can be implemented with the present invention is the ability to paint on a personal computer using a first finger as a brush and a second finger to paint the desired color from a color palette displayed on the screen Select. The first finger is on the surface of the touchpad 904 used to control a conventional primary cursor for painting an image on the computer, which is then on the display monitor 901 is shown. A color palette is implemented in software and on the display monitor 901 is displayed so that the colors can be easily selected. To do this, the user moves a secondary cursor over the desired color in the palette, which is then activated by tapping the touchpad 904 can be selected with the second finger. The user can also use the secondary cursor to change the type or size of the brush in addition to the color using the method described above. This allows the user to paint an image more freely because the user can always hold the primary cursor (brush) over the image.

13th shows schematically a mobile phone 905 with a touchpad sensor 907 according to the present invention. The touchpad sensor is in front of the display 906 provided, wherein the display can still be viewed by the user, while a capacitive coupling between the user's finger or a stylus and the touchpad sensor 907 is made possible. In this example, the user can select and move the items shown on the display using a finger or stylus on the touchpad. The touchpad is integrated with the display so that the "cursor" follows the movement of the finger or pen. So it can have the same functions as above with reference to 12th described for the example of 13th be applied.

Alternatively, the area of the touchpad 907 also be subdivided in such a way that an area of the sensor is only used for the display of the display 906 is used, while another area of the sensor is used only as a conventional touchpad sensor, so that the user can move a cursor and select items in the manner described above.

Another function that can be implemented with the present invention is the control of a game on the mobile phone 905 . Although the touchpad 907 covers the same area as the display 906 , it can be subdivided in such a way that controls are provided to the left and right of a central display area. The user can place two fingers or thumbs on the left and right sides of the touchpad 907 to control two different functions of a game. For example, the left part of the touchpad 907 provide a motion control for moving up, down, left and right while using the right part of the touchpad 907 contains a number of buttons. These can be used, for example, to control a game, with the motion controls on the left side of the touchpad 907 is used for driving a vehicle in a game while using the buttons on the right side of the touchpad 907 used to accelerate or decelerate the vehicle.

It should be noted that the PC and mobile phone described are merely exemplary applications and the invention can be applied to a wide variety of devices such as entertainment devices, telecommunications devices, household devices and car accessories.

From the foregoing, it should be clear that a capacitive touch panel is provided that can detect multiple simultaneous touches that can be input into a gesture state machine in accordance with the present invention.

Claims (14)

A touch sensing device comprising: a sensor having a sensitive area extending in at least one dimension and arranged to output detection signals in response to the proximity of an object to the sensitive area, a position processing unit operable to calculate positions of interactions with the sensitive area from an analysis of the detection signals and to output a time sequence of data indicating the interaction positions on the sensor corresponding to respective touches, and a gesture processing unit which can be operated to analyze the temporal sequence of data and to distinguish one or more gestures from the sequence of data, the gesture processing unit being encoded with a gesture recognition code comprising a plurality of state modules of a state machine, the plurality of state modules includes: a first single-touch state module of the state machine operable to recognize at least one first single-touch gesture and to generate a first output based on the first single-touch gesture, a second single touch state module of the state machine operable to recognize at least one second single touch gesture and generate a second output based on the second single-touch gesture, and a multi-touch state module of the state machine that can be operated, to receive the first output directly from the first single touch state module, to receive the second output directly from the second single-touch state module, and to recognize at least one multi-touch gesture based at least on the first and second outputs, the first single-touch status module, the second single-touch status module and the multi-touch status module being distinguishable, and to output the recognized multiple touch gesture. Device according to Claim 1 , characterized in that each of the plurality of state modules of the state machine including an idle state module and a plurality of gesture interpretation state modules, wherein at the beginning of an operation the idle state module is started and can be returned to by at least some of the gesture interpretation state modules. Device according to Claim 2 , characterized in that each of the plurality of gesture interpretation state modules comprises the state machine including a touch state module for simple touches, the Idle state gives control to the touch state module in response to a touch. Device according to Claim 2 , characterized in that the plurality of gesture interpretation state modules for the multiple touch state module of the state machine including at least one multiple touch state module that is operable to process multiple simultaneous touches based on the first and second outputs from the first single touch -State module of the state machine and the second single-touch state module of the state machine, wherein the gesture processing unit can be operated to control the multi-touch based on the number of simultaneous touches that are defined by the temporal sequence of data at this point in time. To give the state module of the state machine. Device according to Claim 4 comprising: one multi-touch status module for processing the two simultaneous touches and another multi-touch status module for processing the three simultaneous touches. Device according to Claim 3 , characterized in that the plurality of gesture interpretation state modules for each of the first single touch state module of the state machine and the second single touch state module of the state machine including a push state module to which control can be given from a touch state module when a touch with a Duration of more than a threshold duration is detected in the touch state module. Device according to Claim 6 , characterized in that the pressing state is a superstate which comprises several sub-states in order to distinguish between different pressing times. Device according to Claim 2 characterized in that the plurality of gesture interpretation state modules comprises a plurality of state modules operable to recognize movement-related gestures derived from one or more moving touches. Device according to Claim 1 , characterized in that the gesture recognition code is configured to recognize gestures with up to N simultaneous touches, where N is at least 2, and N single-touch state modules of the state machine that can be operated to recognize only single-touch gestures, and N- 1 multi-touch state modules of the state machine that can be operated to recognize only n-way touch gestures, where n = 2 to N. Device according to Claim 1 , characterized in that the position processing unit and the gesture processing unit are accommodated and executed in a single integrated circuit. Device according to Claim 1 , characterized in that the position processing unit is accommodated and executed in a first integrated circuit, while the gesture processing unit is accommodated and executed in one or more separate integrated circuits. Device according to Claim 1 , characterized in that the plurality of state modules comprises a third single-touch state module operable to generate a third output, and in that the state machine is configured to directly display the first, second, and second, from the first, second and third single-touch state modules and receive and third outputs, respectively, and to recognize at least one second multi-touch gesture based at least on the first, second and third outputs and to output the recognized second multi-touch gesture. A method comprising: receiving a time series of data indicating the interaction positions on the sensor corresponding to respective touches, analyzing the time series of data to distinguish one or more gestures from the series of data using a gesture recognition code, comprising a plurality of state modules of a state machine, the plurality of state modules comprising: a first single-touch state module of the state machine operable to recognize at least one first single-touch gesture and to generate a first output based on the first single-touch gesture, a second single-touch state machine of the state machine operable to recognize at least one second single-touch gesture and generate a second output based on the second single-touch gesture, and a multi-touch state machine of the state machine, there s operable to receive the first output directly from the first single-touch state module to receive the second output directly from the second single-touch state module, and to recognize at least one multi-touch gesture based at least on the first and second outputs, wherein the first single-touch status module, the second single-touch status module and the multi-touch status module are distinguishable, and to output the recognized multi-touch gesture. Computer that includes: a memory; a unit operable to output a time series of data indicating the interaction positions on the sensor corresponding to respective touches; analyze the temporal sequence of data to distinguish one or more gestures from the sequence of data using a gesture recognition code comprising a plurality of state modules of a state machine, the plurality of state modules comprising: a first single-touch state module of the state machine operable to recognize at least one first single-touch gesture and to generate a first output based on the first single-touch gesture, a second single touch state module of the state machine operable to recognize at least one second single touch gesture and generate a second output based on the second single-touch gesture, and a multi-touch state module of the state machine that can be operated, to receive the first output directly from the first single touch state module, to receive the second output directly from the second single-touch state module, and to recognize at least one multi-touch gesture based at least on the first and second outputs, the first single-touch status module, the second single-touch status module and the multi-touch status module being distinguishable, and to output the recognized multiple touch gesture.

Images