GB2482559A

GB2482559A - Interacting with touch screen devices via a multi touch device.

Info

Publication number: GB2482559A
Application number: GB1013304.9A
Authority: GB
Inventors: Tom Dudderidge
Original assignee: DISRUPTIVE Ltd
Current assignee: DISRUPTIVE LIMITED
Priority date: 2010-08-06
Filing date: 2010-08-06
Publication date: 2012-02-08
Also published as: GB201013304D0; GB2482571A; WO2012017205A2; GB201104010D0; WO2012017205A3

Abstract

Disclosed of a method of interacting with computing device via itsâ touch screen display panel, the screen being capable of resolving a plurality of contact points. An object with a face is placed against the touch screen, the face has on it a plurality of contact pads, which are able to be recognisable by the touch screen. A software application receives location information from the contact points of the pads. The application then determines details of the object based on the relative positions of the pads, such as data encoded in the pattern of the pads. The pads may be turned conductive and non- conductive by the user as required. The object may have a lower face that is substantially planar, and able to stand on the screen.

Description

Interaction with Touch Screen Devices

FIELD OF THE INVENTION

The present invention is concerned with the range of possible interactions between touch screen devices and the world outside them.

BACKGROUND ART

There are many types of input device for computing systems, including buttons, keyboards, mice, trackballs, joysticks, touch sensor panels, touch screens and the like, Of these, touch screens are becoming increasingly popular because of their ease and versatility 15 of operation, as well as their declining price. Touch screens typically include a touch sensor panel, which can be a clear panel with a touch-sensitive surface, and a display device such as a liquid crystal display (LCD) that can be positioned partially or fully behind the panel so * that the touch-sensitive surface extends over at least a portion of the viewable area of the display device. Touch screens then allow a user to perform various functions by touching * 20 the touch sensor panel using a finger, stylus or other object. In this way, they can interact with a user interface (UI) displayed by the operating system (OS) or application that is controlling the display device.

One design of touch sensor panel typically comprises one or more layers of thin film, deposited and patterned into conductive regions upon at least one layer of a transparent substrate. The conductive regions include a number of capacitive elements arranged into a plurality of rows and columns. When a user's finger contacts a specific region of the touch surface, the approximate location of the user's finger can be determined based upon analysis of one or more sensed signals. Such screens allow "multitouch" functionality, i.e. the independent recognition of more than one "touch event".

A relatively low resolution array of such row and column elements is usually sufficient for finger detection. This is because the width of the typical human finger is relatively large (roughly 10 mm) in relation to at least one dimension of a capacitive element. Therefore, if it is known in advance that the touch sensor panel will primarily be driven by finger input, fewer capacitive elements can be built into the touch sensor panel, and the rows and columns can be separated at a greater distance. A resolution of 4 to 8 mm or so will be adequate.

Another design of touch sensor panel comprises two conductively-patterned sheets separated by a deformable insulating layer. At least the outer conductively-patterned sheet is flexible. Thus, a point contact with the outer layer deforms the layer and the insulating material beneath that point, to bring the conductively-patterned sheets closer together.

This will then affect the capacitance and/or the resistance between the conductively-patterned sheets. By adopting a suitable pattern of conductive regions on the sheets, the location of that touch can be determined.

Generally, touch screens can be overlaid onto any type of standard screen technology (OLED, LED, LCD, etc). The two important subsystems to touch screen technologies are the screen overlay itself (of whatever type) and a controller which processes the variations in electrical properties of the overlay to give useful information about the finger touch(es). S..

S

The signal flow might then be pictured in simplified terms as follows, where the *SISS* * various elements are highly inter-dependent: Device on screen 4 screen overlay 4 controller 4 device OS 4 software application Such panels, whether of one of the above types or a different type, will be referred to herein as "contact-sensitive" displays.

SUMMARY OF THE INVENTION

Contact-sensitive displays are intended to provide an easy and intuitive means for a user to make selections and otherwise interact with an application running on a computing device. We have realised, however, that their use is not limited to this and that they enable a much wider range of interaction with an application than simply detection of a human finger.

In particular, it is possible for items other than a human finger to trigger the display.

Capacitively-sensing displays can be triggered by a reasonably-sized conductive element placed against the display panel so as to cover at least one of the sensor areas. Others can be triggered by a suitably pointed protrusion that is able to create the necessary localised deformation of the screen. Both are exploited in the design of styluses (or styli) which users can employ. However, a stylus does not extend the capability of the device, it merely replaces the user's finger.

The present invention proposes that one or more pads be placed on the underside of an object in order to allow an application running on the computing device to receive data from the object via the display panel. At its simplest, that information could simply be information about the object, including the location and/or orientation of an object such as a play piece for a game illustrated on the display panel. In more complex embodiments, the information could include identification data as to the object, and/or internal state information relating to the object, and/or data held by the object which is to be transferred to the device.

The present invention therefore provides the combination of a computing device having a contact-sensitive display panel capable of resoLving a plurality of contact points on the display panel, a software application loaded onto the computing device, adapted to I..

receive location information as to the plurality of contact points; and an object having a face placeable against the display panel on which is defined a plurality of contact pads adapted to be recognisable by the display panel, wherein the application is adapted to receive data from the object encoded in the location information as to the plurality of contact points.

The data received by the application can include details of the object, such as orientation data relating to the object, data relating to the nature of the object, and/or data relating to the identity of the object. Data from the object can be encoded in some or all of the relative positions of the pads, the strength of the signal detected by the display panel, and a variation with time of the signal detected by the display panel, or otherwise. The application can determine the details of the object by comparing the location information as to the plurality of contact points to a database of object details, although preferably the application will normalise the location information as to the plurality of contact points prior to comparison.

The pads are preferably in a pattern with limited or no rotational symmetry, and can comprise a bump on the face of the object and/or a conductive region on the face of the object. In the latter case, the pads can be selectably conductive, for example becoming conductive when the object is activated by a user such as through use, touch and/or movement.

The object preferably has a lower face that is substantially planar, thereby enabling it to stand in a stable and unsupported manner on the display.

The present invention also relates to an object for placing on a contact-sensitive display panel, comprising at least one pad adapted to be recognisable by the display panel.

Such an object is ideally stable standing in an upright state on the panel. Other preferred features of the object are as defined above.

BRIEF DESCRIPTION OF THE DRAWINGS

An embodiment of the present invention will now be described by way of example, with reference to the accompanying figures in which; * * Figure 1 shows a first embodiment of the invention, from above; 0., )

S SS

Figure 2 shows the first embodiment from one side;

S

Figure 3 shows an alternative version of the first embodiment, from one side; S,*b ** S 4 Figure 4 shows an arrangement for an active pad; Figure 5 shows an alternative arrangement for an active pad; Figure 6 shows a second embodiment of the invention, from above; Figure 7 shows the second embodiment, from one side; Figures 8 to 13 show pad arrangements for the second embodiment; Figure 14 shows a third embodiment of the invention, from above Figure 15 shows the third embodiment from one side; and Figure 16 shows an alternative design of an active pad.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Figures 1 and 2 show a first and simplest embodiment of the present invention.

Figure 1 shows the contact-sensitive multitouch display panel 10 in plan view (i.e. in the direction normally viewed by a user). Lighter lines 12 delineate the square grid of touch-sensitive elements within which a touch by a user will be recognised as a "touch event" and passed to the application running on the device.

A larger grid 14 is displayed on the screen and visible to the user, this forms a chequerboard pattern with a linear spacing of (in this case) three times the touch-sensitive grid. In practice, alternate diagonal squares of the chequerboard will be shown in a contrasting colour in order to contrast with the remaining squares, and the chequerboard grid will ideally match closely to the touch-sensitive grid. However, for illustrative purposes the shading is omitted and the grids are shown slightly displaced so that both can be discerned. *..S

I . *.

: A user can then place one or more chequer pieces 16 on the display. Each consists of a solid disc 18, on the underside of which is a conductive pad 20 that is capable of Is. j activating the contact sensors of the display panel 10. The discs 18 are sized to fit neatly * 20 within one of the chequerboard squares 14, and the pad 20 is about half the diameter of the disc 18 and therefore comfortably covers a square of the touch-sensitive grid 12.

I. *IS5 * I *e Thus, the application can be provided with accurate positional data regarding the discs 18 by way of their interaction with the display panel 10. This enables a game of chequers (also known as checkers, or draughts) to be played intuitively. The player can move his or her pieces across the board that is displayed on the display panel and the application will be able to detect those moves.

This permits a number of styles of gameplay. Two real persons can play, using the screen as the board and with the application checking the legality of moves and alerting the players appropriately when an illegal move is made. A real person could play against the application, with the application's virtual pieces being displayed digitally on the display panel and an Al engine of the application determining the device's next move. A real person could also play against a remote real person, with both people using compatible devices and each device displaying a set of virtual pieces corresponding to the remote person's physical pieces.

In other variations, the relative proportions of the two grids could be changed. Each chequerboard square could correspond to a single touch-sensitive pixel, or to more than 9 pixels.

A variant form of the physical game piece 16' is shown in figure 3. The pad 20' is surrounded by an annular trim region 22 that is of the same thickness as the material of the pad 20' and therefore gives the piece 16' a smooth lower surface. Alternatively, the pad 20' could be recessed into the disc 18'.

Some contact-sensitive multitouch display panels are only able to report a limited number of touch events. For example, applications running on the Apple iPad are given access to a maximum of 10 simultaneous touch events, although it is thought that the operating system is able to distinguish more. This restriction will no doubt be relaxed in due course as the technology develops, but in the meantime it may prove to be a limitation. To avoid this, the game could be limited accordingly, or the play pieces could be selectively activated, for example when touched or moved. Thus, when the play piece senses that it is 0 being touched, the pad 20 becomes conductive substantially immediately, and is detected by * the application. The application now knows that the play piece is being touched and may move. If it disappears from the contact-sensitive screen and a new touch event happens on a square that did not previously have a play piece, then the application knows that the play piece has been moved to that square. This could also allow the concept to be applied to * 5 non-multitouch screens. *.*

This could be achieved by providing semiconducting areas on the underside of the play piece. Alternatively, a thin conductive layer could be provided which is (on its own) unable to trigger the contact-sensitive screen but which is selectively connectable to a more substantial conductive element, with the combined capacitance of the two being sufficient to trigger the screen. The selective connection between the two could be active, or it could be passive by way of conductive tracks on the surface of the play piece which are bridged by a (conductive) finger. In this way, a touch-sensitive element (in the form of the play piece) decides when to activate the touch-sensitive screen.

Figure 4 shows another way of providing a selectively conductive (or "active") pad 20. The circular pad is divided into several segments, in this case four segments 20a, 20b, 2Cc, 20d. The total number of segments can be decided on so that the individual segments do not present sufficient capacitance to trigger a touch event, but the segments when connected do present sufficient capacitance. Thus, wires or conductive tracks 24, 26, 28, 30 or the like lead from each lead from one pad 20a-20d to a switch 32. This has a conductive element 34 which is able to bridge all the conductors 24, 26, 28, 30 when the switch is activated and (electrically) join the pad segments 20a-20d. The conductive element 34 can be a part of the switch or it can be the surface of a user's finger, which will usually have sufficient conductance.

Figure 5 shows an alternative arrangement. A number of nesting conductive spiral tracks (in this case, two) 36, 38 define the pad 20. Each leads to a switch 40, allowing the two tracks to be joined thereby increasing the capacitance of the pad 20 above the threshold necessary to trigger a touch event.

The various devices available on the market at present include a wide variety of contact-sensitive displays, each of which have such a wide variety of resolutions and sensing mechanisms. It is therefore likely that that various active pad arrangements described 0 above will work on only some of the available devices but not on others. *S... * *

Figures 6 and 7 show a development of the first embodiment, allowing a game of chess to be played. This involves the additional complexity that the play pieces are non-identical, consisting of two rooks (castles), knights and bishops, one queen, one king, and : eight pawns for each player. Thus, the application needs to distinguish between these. One * 25 way of doing so is by remembering; provided that the play pieces are set out in the correct *** pattern at the start of the game, the application can thereafter follow which moves are made and therefore remember which piece is in which location. However, this may be vulnerable to set-up errors and is not resilient to later mistakes.

Figures 6 and 7 therefore show a second embodiment, adapted for a game of chess (or similar board games). In this example, a single square 50 of the chequerboard pattern corresponds to a larger number of touch-sensitive pixels 52 on the display 54, in this case 7x7 but other relative sizes are possible. Clearly, this requires the overall play area to be larger or the individual pixels to be smaller.

This allows the play piece 56 to be provided with a number of conductive pads 58, 60, 62 rather than just one as in the first embodiment. As in the first embodiment, these can be selectively conductive if required, such as to limit the total number of simultaneous touch events with which the device will need to cope. Such a plurality of pads allows different types of play piece to be distinguished from each other and their identity confirmed.

This is illustrated in figures 8 to 13. Figure 8 shows a simple pattern suitable for the commonest pieces, the pawns 64, which consists of one central pad 66. Figures 9 and 10 show more complex patterns for the king 68 and queen 70, each of which includes 3 pads arranged in a distinctive pattern. Thus, the three pads 72, 74, 76 beneath the king piece 68 are arranged in a straight line; one pad 74 at the centre of the piece 68 and the other two pads 72, 76 on either side. The queen 70 has three pads 78, 80, 82 arranged in a triangular pattern.

As a result, if the device detects only one pad 66, it can conclude that the square 50 is occupied by a pawn 64. If it detects three pads then it knows the square 50 is occupied by a king or queen; if the three pads 72, 74, 76 are aligned then the piece must be the king ?0 68 whereas if they are not aligned then the piece must be a queen 70.

S..... * *

Figures 11 to 13 show possible patterns for the other chess pieces, the rook 84, : knight 86 and bishop 88 (respectively). Each of these has two pads, arranged in unique patterns. To allow for possible rotation of the play piece and for possible offset positioning : of the piece within the square 50, these rely on the distance between the pads. Thus, the * *25 rook 84 has two widely-spaced pads 90, 92 with approximately three touch-sensitive pixels S..

between them. The knight 86 has two intermediately-spaced pads 94, 96, spaced by about two touch-sensitive pixels, and the bishop 88 has two closely-spaced pads 98, 100, spaced by a single touch-sensitive pixel.

The allocation of pad patterns to particular pieces is of course entirely arbitrary. All that is needed is sufficient unique patterns to cater for the number of unique pieces in the game. Generally, if there are more of some pieces than others then it may be advantageous to alloacte patterns with fewer pads to those pieces so that the total number of touch events that need to be tracked can be minimised. Equally, more patterns could be employed so that (for example) individual pawns could be distinguished from each other.

This might require a greater number of pads, a greater number of pad patterns, or the use of a time-or intensity-varying pad or pads.

With the chess pieces thus identifiable and locateable on the board, the application running on the device can play, supervise or enable chess games as described above in relation to draughts. Thus, a user can play against another user, or against the computer, or against a remote user.

Based on the above concept, a wide variety of games can be embodied. As well as traditional board games, other board games or any piece-based game can be implemented.

Playing pieces could be distributed separately to the game software, for example as free gifts with other purchases such as fast food, or as collectable sets of cards or toys. Different pieces or cards could be provided with a different pattern of pads that can be interpreted by the game software accordingly, such as to enable aspects of the game or gameplay thereby creating a perceived value for the piece.

Figures 14 and 15 show a third and further embodiment of the present invention, in which the use of active pads such as are described above and illustrated in figures 4 and 5 is * O extended to allow for greater interaction between the play piece and the display. In this * case, the play piece 102 resting on the display panel 104 is in the form of a small or handheld electronic device, of a form factor similar to that of a Tamagotchi®, mobile telephone, or mp3 player.

The play piece 102 is itself a computing device, able to run an application or * 25 embedded software that may provide a game or other function, controlled via buttons 106, **S 108 and displayed on a small screen 110. For example, the well-known Tamagotchi® device allows a user to interact with a virtual pet, and depending on the user's actions the pet may thrive, grow, multiply, or become ill and/or die. Different games (or other software) will generally show an input-related development over time. Thus, the data retained by the play piece will change over time.

To allow this development to be communicated from the play piece 102 to the device, the play piece has a plurality of active pads 112 on its underside, and therefore in contact with the display panel 104. These are arranged in an array that is spaced to correlate with spacing of contact-sensitive pixels 114 of the display panel 104, and are sized to match one pixel 114. For clarity of illustration, in this embodiment the pads 112 are spaced at twice the pitch of the pixels 114 and thus each pad 112 has a clear pixel 114 between it and the adjacent pad. To assist the user in correct placement of the play piece 102 on the display panel, guidelines 116 are shown on the display 104. These can correlate with the external shape of the play piece 102. Alternatively, an image of the play piece 102 could be shown on the display 104.

In other embodiments, the pads may be more closely spaced so as to use the available pixels more efficiently, so that there is a 1:1 relationship between the pixels 114 and the pads 112. Equally, a greater or lesser number of pixels could be provided; the 4x4 grid of 16 pixels shown in figure 14 will allow for 16-bit communication between the play piece 102 and the device.

The active nature of the pads 112 allows them to be controlled by the play piece so as to choose a pattern of conductive pads 106 that sends a specific message to the device.

For example, with 16 pads 112 as shown, this allows over 60,000 different static states to be communicated. In theory, the upper limit is 2161, if we assume that the null state is to S...

!0 be discounted as this is indistinguishable from the absence of the play piece. The actual : total may be less than this, depending on what assumptions we make. For example, an upper limit of 21617 applies if we also discount single-pad states which may be somewhat : more difficult to identify reliably. Alternatively, reducing the upper limit to 2121 allows us to use the four corner pads as reference points for locating the other 12 pads, which reduces the total number of states to just over 4,000 but may be more reliable in practice. This may * be adequate for a wide range of devices; if not, then either more pads 112 could be *** provided and/or the state of individual pads could be made time-dependent.

Generally, n' usable active pads can adopt up to 2'-l states, and n' pads with a state that is variable with a time period of t'will be able to communicate about (2-1)T/t states over a time T. There is some approximation in this latter figure in that it may be possible to use the null state for intermediate states (i.e. other than the first and last state).

In this way, the play piece 102 can communicate data to the device via the contact-sensitive screen. This allows the result of prior inputs to the play piece 102 to be passed to the device easily, just by placing the play piece on the device display. The play piece 102 shown in figures 14 and 15 can also receive data from the device via a photo-sensitive element 118 which is positioned so as to view a part of the display. This permits the device to receive a time-varying signal sent by illuminating the appropriate pixels of the display panel. With two-way communication of this type, handshake and error correction protocols also become possible. Alternatively, other forms of wireless communication could be used instead of or in addition to the above, such as Bluetooth, WiFi, etc. The active pads 112 of the play piece 102 of the third embodiment can be as described above in relation to figures 4 and 5, or they may be as illustrated in figure 16.

This shows a section through a pad 112. A thin conductive film 120 is provided at the base of the pad, perhaps covered with a protective layer (not shown). This is covered with an insulating layer 122, and above that is a larger conductive mass 124. A semiconductor switch 126 is provided between the two conductive elements 120, 124 and is connected to each via wires or conductive tracks 128, 130. The switch 126 may be provided within the pad 112 (as illustrated) or elsewhere within the play piece 102.

To activate the pad 112, the switch 126 connects the two conductive elements 120, 124. This causes the effective capacitance of the thin film 120 to increase. Through careful *S.S design of the thicknesses of the various conductive and insulating layers 120, 122, 124 it is possible to ensure that this increase in capacitance takes the capacitance of the pad 112 from under the threshold for triggering a touch event to over the threshold.

A range of other types of object are also possible. For example, a doll-type toy could include sufficient pads (or any desired type) for the device to recognise its orientation :25 and/or position, stance, pose etc. This doll could then interact with the application running on the device.

Thus, the invention provides a range of types of pad, including: Passive' pads, normally arranged in multiples, arranged to communicate the identity and/or the position of an object, Active' pads, which are selectively active so as to transmit state information and the like (such as, that a chess piece is being or about to be moved) etc, and Smart' pads which can communicate data through the touchscreen from the object to the computing device (and vice-versa, if combined with photosensitive or like devices).

As noted above, the two main types of contact-sensitive screens are resistive touch screens and capacitive touch screens. There is active ongoing debate about the relative pros and cons of resistive vs. capacitive touch screen technologies. Generally, there are price advantages to resistive screens, versus the multi-touch capabilities of capacitive screens. As a result, there is no clear agreement about the importance of multi-touch and how this will evolve. In the above, we refer to a plurality of contact points which is most easily embodied in a multi-touch screen, which will usually be capacitive. However, the principle is applicable to resistive screens either by adopting a more limited form of application (such as the draughts/checkers game described above) or by using other means to inform the device as to the identity of the object, such as encoding the identity data via a a time-varying signal or in the signal strength, or a combination of both.

Likewise, we describe a matrix of sensors for the touch-sensitive screen, In practice, there are several different physical formulations for the overlay in each case (resistive and capacitive). These vary from matrices of sensing elements to continuous (analogue) measurement systems. *.

2O For example, a mutual capacitance screen is a true matrix of sensing elements as described above. Screens based on "Self capacitance" measure along vertical and horizontal lines, which can be thought of as a matrix grid of sensing elements. A 4-wire resistive screen, on the other hand, has a continuous voltage gradient across the x and y axes. In that case, the screen does not have pixel elements as such but does report a location.

25 Thus, we can cause the object to vary four types of variables on a touch screen in order to determine the location and identity of an object placed on the screen. These are the location, orientation, time and strength of the signal.

It will of course be understood that many variations may be made to the above-described embodiment without departing from the scope of the present invention. In particular, it is to be expected that the abilities of contact-sensitive displays will develop over time. Linear developments such as improvements in resolution and improvements in the number of discrete touch events that can be recognised simultaneously will extend the complexity of the above-described embodiments in a corresponding manner. Other developments will allow more varied adjustment of the embodiments; for example, future contact-sensitive displays may be able to detect different shapes and sizes of capacitive contacting objects which will permit the information about the object to be encoded using shape and size rather than merely pixel number and position. Displays may also be triggered by properties other than capacitance, leading to corresponding adjustment of the nature of the contacting pads. * *

S

555*5s * S * S S ** *

S *S.

I

5.55..

I

S *15 *

Claims

CLAIMS1. The combination of; a computing device having a contact-sensitive display panel capable of resolving a plurality of contact points on the display panel; a software application loaded onto the computing device, adapted to receive location information as to the plurality of contact points; and an object having a face placeable against the display panel on which is defined a plurality of contact pads adapted to be recognisable by the display panel; wherein the application is adapted to receive data from the object encoded in the location information as to the plurality of contact points.
2. The combination according to claim 1 in which the received data consists of information concerning the object.
3. The combination according to claim 2 in which the information concerning the object includes orientation data relating to the object.
4. The combination according to claim 2 in which the information concerning the object includes data relating to the nature of the object.
5. The combination according to any one of claims 2 to 4 in which the information concerning the object includes data relating to the identity of the object.
6. The combination according to any one of the preceding claims in which data from * the object is encoded in the relative positions of the pads.
7. The combination according to any one of the preceding claims in which data from * the object is encoded in the strength of the signal detected by the display panel.S
8. The combination according to any one of the preceding claims in which data from the object is encoded in a variation with time of the signal detected by the display panel.
9. The combination according to any one of claims 1 to 6 in which the application determines details of the object by comparing the location information as to the plurality of contact points to a database of object details.
10. The combination according to claim 9 in which the location information as to the plurality of contact points is normalised by the application prior to comparison.
11. The combination according to any one of the preceding claims in which pads are in a pattern with limited rotational symmetry.
12. The combination according to any one of claims 1 to 10 in which the pads are in a pattern with no rotational symmetry.
13. The combination according to any one of the preceding claims in which the pads comprise a bump on the face of the object.
14. The combination according to any one of the preceding claims in which the pads comprise a conductive region on the face of the object.
15. The combination according to claim 14 in which the pads are selectably conductive.
16. The combination according to claim 15 in which the pads are selectably activated by a user.
17. The combination according to claim 15 in which the pads are made conductive when * St...* the object senses use. *SS*

:
18. The combination according to claim 15 in which the pads are made conductive when 20 the object senses touch.
19. The combination according to claim 15 in which the pads are made conductive when the object senses movement.
20. The combination according to any one of the preceding claims in which the object has a lower face that is substantially planar.
21. An object for placing on a contact-sensitive display panel, comprising at least one pad adapted to be recognisable by the display panel, the object being stable standing in an upright state on the panel.
22. An object for placing on a contact-sensitive display panel, comprising a plurality of pads adapted to be recognisable by the display panel.
23. An object according to claim 22, being stable standing in an upright state on the panel.
24. An object for placing on a contact-sensitive display panel substantially as any one herein described with reference to and/or as illustrated in the accompanying figures. *.S. * * S. S* ** S. SS* * S * . S *5 S S..SS..... S 0S