GB2464848A

GB2464848A - Real-time digital pen stroke display

Info

Publication number: GB2464848A
Application number: GB0922454A
Authority: GB
Inventors: Mark Peter Bird; James Gardiner; Edward Oscar Belgeonne
Original assignee: DESTINY WIRELESS PLC; STELJES Ltd
Current assignee: DESTINY WIRELESS PLC; STELJES Ltd
Priority date: 2007-01-10
Filing date: 2007-01-10
Publication date: 2010-05-05
Anticipated expiration: 2027-01-10
Also published as: GB2446203A; GB0922454D0; GB2446203B; GB2464848B; GB0700463D0

Abstract

A digital pen interaction system includes a computer, a digital pen, and a position-coding writing surface. A user operates the digital pen to generate pen stroke data corresponding to positions of a writing tip of the pen on the position-coding writing surface when the digital pen senses a writing event. This pen stroke data is transmitted wirelessly in real time to the computer. The computer has a display and receives the pen stroke data wirelessly in real time from the digital pen and processes the pen stroke data. The display displays a graphical representation of the pen stroke data in real time.

Description

SYSTEM AND METHOD FOR OPERATING A DIGITAL PEN

The present invention relates to a system and method for operating a digital pen. Particularly, but not exclusively, the present invention relates to the interaction between a writing surface, a digital pen and a computing means.

It is known to use a writing tablet and a stylus for inputting hand written data into a computer or other information processing system. The tablet typically has a complex structure, since it is required to co-operate with the stylus to determine electronically the relative position of the stylus on the tablet. Furthermore, the movement of the stylus over the surface of the tablet, whilst providing data for input into an associated computer, does not provide any change in the surface of the tablet itself that is visible to the user.

WO 97/22959 discloses an apparatus for absolute optical position determination that is radically different from the tablet and stylus arrangement outlined above. Rather than requiring a tablet, WO 97/22959 discloses the use of specially printed paper and a pen that, as well as writing on the paper, can determine the position of the pen with respect to the paper and can therefore generate data corresponding to the markings of the pen on the paper. The pen includes a stylus or nib that contacts the paper to activate the optical detection system of the pen. The pen communicates with other devices via a wireless transceiver.

WO 00/72230 discloses a similar system to WO 97/22959 except that the pen of WO 00/72230 includes a motion sensing device such as an accelerometer to periodically sample the acceleration of the pen so as to generate movement data. It is intended that the system of WO 00/72230 is used with paper printed on a netpage printer, so that a user can interact with a netpage to fill in forms, for example.

The technology of WO 97/22959 has been developed by the company Anoto AB, Sweden. Anoto have produced a proprietary pattern of very small dots printed on paper. The pattern of dots on white paper is perceived by the unaided human eye as having a slightly off-white colour. The pattern of dots is such that only a very small area of the total pattern needs to be imaged in order for there to be a unique determination of the position in the full pattern. The total area of the full pattern is about 60 million square kilometres.

The pattern of dots interacts with the imaging device of suitable digital pens to allow information coming from the pen to be processed into signals representing functionality, writing and drawing.

The Anoto pattern provides small dots with a nominal spacing of 0.3 ruin (0.01 inch). These dots are slightly displaced from a grid structure to form the proprietary Anoto pattern.

When writing with a digital pen on paper printed with the pattern, digital snapshots of the pattern are typically taken with a rate of more than 50 per second. Every snapshot contains enough information to make a calculation of the exact position of the pen. The intelligence in the paper, derived from the pattern, makes it possible to perform operations by ticking a box with the pen, for example to send a form.

One of the main advantages of the Anoto system is that the user is able to use the pen in a normal way, writing or drawing on paper so as to have a hard copy of the writing or drawing.

WO 2006/049573 discloses a digital pen suitable for use with Anoto pattern paper. This document explains that the pen captures a series of images of the surface being written or drawn on. These images are processed to generate a sequence of data items, typically one position for each image. These positions are then continuously input to an offline process or an online process, dependent on a selection made by the user. In the offline process, the data items are stored in a persistent memory in the pen. In the online process, the positions are buffered in a temporary memory before being output from the pen.

WO 2006/049573 explains that the pen may store one or more templates that define how the pen is to operate on the information that is recorded from different parts (such as functional areas) of the position-coding pattern. Such functional areas may be denoted as "pidgets". One suggested function for a pidget in WO 2006/049573 is a streaming attribute, which indicates to the pen that the recorded positions falling within the area of interest should be output in real time to an external device. However, the default operation of the device of WO 2006/049573 is to store the recorded positions in the pen memory as offline data, these only being downloaded to an external device at a later time.

Various companies other than Anoto AB provide digital pens for use with Anoto functional paper. One such company is Logitech Europe S.A., Switzerland. Logitech's published patent application WO 2005/010634 describes a digital pen that is capable not only of capturing normal pen strokes (such as handwriting and drawings) but is also capable of carrying out functional operations on such strokes (e.g. highlighting) by operation of function control buttons on the pen. The content of WO 2005/010634 is hereby incorporated by reference in its entirety.

Various communications interfaces are possible for the devices set out in WO 2006/049573 and WO 2005/010634, such as USB or RS232 (wired communication), or radio transmission, ultrasound transmission, infrared transmission, inductive coupling, etc. (wireless communication) One particular short-range wireless communication protocol is widely known as Bluetooth (registered trade mark) Bluetooth technology operates in the unlicensed industrial, scientific and medical (ISM) band at 2.4 to 2.485 GHz, using a spread spectrum, frequency hopping, full-duplex signal at a nominal rate of 1600 hops/sec. The 2.4 GHz ISM band is available and unlicensed in most countries Bluetooth is also known as IEEE 802.15.1. The core specification (Core Specification v2.0 + EDR, published 4 November 2004) for Bluetooth is available at http://www.bluetooth.com (accessed 8 January 2007) The content of this core specification is hereby incorporated by reference in its entirety.

WO 2005/010634 suggests that the digital pen described therein can communicate with a dock via Bluetooth. This occurs when the pen is located at the dock, and is typically an offline dump of data from the pen to an information processing device (such as a personal computer) via the dock.

This is the typical mode of operation of Anoto-functionality pens on the market at the time of writing.

The present inventors have realised that it would be advantageous to allow data corresponding to pen strokes on a position-coding surface to be streamed continuously and wirelessly into a graphical display software application.

Accordingly, in a first aspect, the present invention provides a method of operating a digital pen interaction system, the system including: a computing means; a digital pen; and a position-coding writing surface, wherein: the digital pen is operated to generate pen stroke data corresponding to positions of a writing tip of the pen on the position-coding writing surface when the digital pen senses a writing event, and said pen sends said pen stroke data wirelessly substantially in real time to said computing means; the computing means has at least one associated display means and receives said pen stroke data wirelessly substantially in real time from said digital pen and processes said pen stroke data to be displayed on said display means via a graphical display software application substantially in real time.

In a second aspect, the present invention provides a computing means operatively programed to carry out the method according to the first aspect.

In a third aspect, the present invention provides a computer program capable of carrying out the method according to the first aspect when loaded onto a suitable computing means.

In a fourth aspect, the present invention provides a storage medium such as an optical or magnetic disk, or a solid state storage medium, storing the computer program according to the third aspect.

In a fifth aspect, the present invention provides a kit including: a computer program according to the third aspect or a storage medium according to the fourth aspect; a position-coding writing surface; and a digital pen capable of generating pen stroke data corresponding to positions of a writing tip of the pen on the position-coding writing surface when the digital pen senses a writing event, and capable of sending said pen stroke data wirelessly substantially in real time to a corresponding computing means.

In a sixth aspect, the present invention provides a digital pen interaction system, the system including: a computing means; a digital pen; and a position-coding writing surface, wherein: the digital pen is operable to generate pen stroke data corresponding to positions of a writing tip of the pen on the position-coding writing surface when the digital pen senses a writing event, and operable to send said pen stroke data wirelessly substantially in real time to said computing means; the computing means has at least one associated display means and said computing means is operable to receive said pen stroke data wirelessly substantially in real time from said digital pen and to process said pen stroke data to be displayed on said display means via a graphical display software application substantially in real time.

It will be understood that the requirement here for real-time operation is as perceived by a human operator or observer. Thus, it is envisaged that the pen stroke data may be sent between the pen and the computing means via any convenient packet-based communications protocol, provided that the resultant updating of the image shown on the display means via the graphical display software application is substantially without perceivable delay. For example, an acceptable delay may be two seconds or less, and is preferably significantly less than this. For example the delay may be 1.5 seconds or less, 1 second or less or 0.5 seconds or less. In cases where the system operates to convert handwritten text, characters or symbols into ASCII characters for display on the display means via the graphical display software application, there may be a small additional delay. This is typically because it is necessary at least for the user to complete a handwritten character or symbol (or a series of such characters or symbols) before it or they can be interpreted and converted into the appropriate ASCII character or characters. In that case, a delay of up to 2-4 seconds may be acceptable, although of course a reduced delay is desirable.

Preferably the writing surface is a markable writing surface.

It may be a permanently markable writing surface such as a paper writing surface. Preferably the position coding of the surface is provided by a suitable pattern applied to (e.g. printed on) the surface. The position coding may be provided by at least one part of the Anoto (registered trade mark) proprietary pattern. Most preferably this part of the pattern includes instructions for the digital pen to operate in a wireless streaming mode.

Preferably the writing surface includes a first active area.

Preferably the first active area occupies a major portion (i.e. 50% or more) of the writing surface. Preferably the pattern of the first active area, when sensed by the digital pen, instructs the digital pen to operate in wireless streaming mode. ost preferably the pattern of the first active area instructs the digital pen to operate to send pen stroke data in wireless streaming mode in real time.

Preferably at least part of the perimeter of the first active area is marked so that the user can identify the perimeter of the first active area. This is of use so that the user can determine whether or not a particular pen stroke made on the writing surface will be transmitted to the computing means.

The writing surface may include additional active areas, separate from the first active area. The preferred functionality of the additional active areas is discussed below.

Preferably the digital pen includes an ink reservoir. In this way, the writing tip of the digital pen can be operated by the user to write or draw in a normal fashion on the writing surface, to apply visible marks to the writing surface. In this way, the user can be provided with a hard original of the markings applied to the writing surface, and what is displayed on said display means via the graphical display software application can be edited, saved or deleted as desired in the normal way on the computing means.

Preferably the computing means has an integrated wireless transceiver operable to pair with a corresponding wireless transceiver of the digital pen. It is possible for the wireless transceiver of the computing means to be a stand-alone module, for example connected to the computing means via a wired link, but this is not preferred since it leads to a more complex arrangement for the system. Preferably the wireless transceivers of the computing means and the digital pen operate according to a standardised wireless communications protocol. A suitable wireless communications protocol is Bluetooth (registered trade mark), which is also known as IEEE 802.15.1, and preferably is based on the core specification of that protocol (Core Specification v2.O + EDR, published 4 November 2004) In addition to the hardware wireless transceiver, the computing means typically also runs software (or drivers) to interact with the transceiver in order to handle the communications protocol. Such software is typically partitioned in modules, or layers, according to the OSI (Open Systems Interconnection) network model, and thus is typically referred to as a stack.

Preferably the computing means operates via a Windows (registered trade mark) based operating system. The computing means may be a personal computer such as a desktop PC or (more preferably) a laptop PC or other mobile computing means. It is also possible for the computing means to operate on a different operating system, such as Linux or Apple Mac OS (registered trade mark) Preferably the graphical display software application is suitable for the preparation of electronic display slides.

Most preferred is that the graphical display software application is Microsoft (registered trade mark) PowerPoint (registered trade mark) . It is preferred that the graphical display software application has an operating mode in which what is displayed at the display is only the visible content of one or more electronic display slides. In PowerPoint at least, and in other similar applications, such an operating mode is known as "view slide show" mode. This is intended for presenting the content of the slides to an audience of one or more persons. For example, it is particularly suitable for making presentations (e.g. teaching to a classroom or lecture room) on a large format display, such as on a projected display. Thus, in "view slide show" mode, it is typically not possible to edit the content of the slide or slides without exiting "view slide show" mode.

Such editing normally is carried out in "edit/create slides" mode in such applications, in which the user is presented with, in addition to a preview view of the slide or slides, various functional buttons and tabs to assist with entering and formatting content in the slides. It will be appreciated that the reason why the "view slide show" mode does not display these additional functional buttons and tabs is to avoid detracting from the overall visual impact of the content of the slides.

Preferably there is provided on the computing means or storage medium a plug-in for interfacing with the graphical display software application. Most preferably the plug-in provides a data receiving and processing interface between the wireless communications stack and the graphical display software application.

The plug-in may provide parsing means for receiving pen stroke data from the wireless communications stack.

Preferably the pen stroke data is received by the plug-in in native Anoto format. This pen stroke data is then preferably processed in order to provide interpreted pen stroke data, typically in a form that is directly insertable into the graphical presentation display software to provide inserted pen stroke data (i.e. "inserted pen stroke data" is intended to mean an image displayed in the graphical presentation display software corresponding to (and optionally an interpretation of or conversion of) the pen strokes made by a user on the writing surface using the digital pen) It is possible for the plug-in to provide one or more further processing of the interpreted pen stroke data.

One further processing of the interpreted pen stroke data is for the plug-in to provide means for passing the interpreted pen stroke data to handwriting conversion means. The handwriting conversion means may for part of the plug-in, but alternatively may be separate. The handwriting conversion means typically includes one or more databases (libraries) for conversion of handwritten characters or symbols into ASCII characters and typical word processing functions. Such handwriting conversion means are known to the skilled person.

Another further processing (which may be additional or alternative) is for the plug-in to provide further interpretation of the pen stroke data in order for further characteristics to be determined so that these characteristics can be applied to what is to be displayed by the graphical presentation display software. For example, the further processing may affect the colour of one or more lines to be displayed. As another example, the further processing may affect the thickness of one or more lines to be displayed. As a further example, the further processing may affect whether the lines being drawn by the user are to be grouped. As a still further example, the further processing may affect whether the lines are to be inserted into the graphical presentation display software according to their absolute positions (typically known as "free draw") or whether the lines are to be inserted according to a position approximation protocol (typically known as "snap draw" or "snap grid") It is preferred that the above described additional functions provided by the plug-in are selectable by the user.

They may be selected by appropriate interaction of the digital pen with the writing surface. In particular, the writing surface may provide second and subsequent active functional areas (e.g. pidgets) corresponding to the required functions. In a preferred mode of operation, activation of one of these pidgets (e.g. by touching the digital pen writing tip at the pidget) is communicated to the computing means and is interpreted by the plug-in to instruct that the subsequent pen strokes in the first active area are to be processed according to the function of the pidget (e.g. ASCII conversion, line colour, line thickness, etc.) It is of course possible that the user will write the required characters and symbols at a deliberately chosen size, e.g. for emphasis or visibility to an audience. This is particularly the case in a teaching environment. It is preferred that, if the system operates to convert such characters and symbols to digital text, that the plug-in provides text size matching means so that the font size of the digital text (ASCII characters) shown in the display scales with the chosen size of the handwritten characters or symbols.

It is preferred that the data output from the plug-in is inserted to the slide or slides when the graphical display presentation application is in "edit/create slide" mode.

However, in one preferred mode of operation, it is also possible for the data output from the plug-in to be inserted into the slide or slides during "view slide show" mode without the need for the application to exit "view slide show" mode. This has particular relevance where the system is used in a presentation environment (e.g. teaching or lecturing) so that the user (or users) can input data into the displayed slide without the need also for the display of the various elements associated with the "edit/create slide" mode of the graphical display presentation application.

The plug-in may provide means to allow more than one digital pen to interact with the graphical display presentation software. In a single-pen system, typically operation of the pen in one of the active areas of the writing surface causes a registration event for that pen with the plug-in.

Subsequently, as discussed above, pen stroke data from a first such pen is typically inserted into a document (or slide) of the graphical display presentation software as inserted pen stroke data. Preferably the inserted pen stroke data from the first pen has a first characteristic applied to it. This first characteristic may be a first colour. Additionally or alternatively the first characteristic may be a first label. The intention here is for the first characteristic to identify the fact that the inserted pen stroke data has originated from the first digital pen. Pen stroke data from a second digital pen may be inserted into the same or a different document (or slide) of the graphical display presentation software. This typically follows a second pen registration event with the plug-in. Preferably (and particularly where the pen stroke data from the second pen is inserted into the same document (or slide) as the pen stroke data from the first pen), the inserted pen stroke data from the second pen has a second characteristic applied to it, in order to distinguish it from the inserted pen stroke data from the first pen. This second characteristic may be a second colour. Additionally or alternatively the second characteristic may be a second label. The first and second digital pens may be operated by the same user. In that case, it is preferred that the first and second digital pens are used on the same writing surface, e.g. physically the same piece of paper. It is especially preferred in this case that the first and second digital pens themselves have ink of differing colours, so that the pen strokes applied to the paper may also be differentiated as originating from different pens. The ink of the first pen may correspond in colour to the characteristic colour of the inserted pen stroke data from the first pen. Similarly, the ink of the second pen may correspond in colour to the characteristic colour of the inserted pen stroke data from the second pen.

It is possible that the first and second pens may be operated by different users. In that case, there may be provided a first writing surface for the first pen and a second, different writing surface for the second pen. In this case, it is possible for the ink colour of the first pen to be the same as the ink colour of the second pen. In the case where there are two writing surfaces, they may have different position-coding patterns or they may have substantially identical position-coding patterns. The advantage of providing substantially identical position-coding patterns is that the relative positions of pen strokes made on the different writing surfaces is more easy to correlate.

Where the pens are operated by different users, the inserted pen stroke data from the first pen may be displayed or displayable in the same document (or slide) of the graphical display presentation software. Alternatively the inserted pen stroke data from the first pen may be displayed or displayable in a first document (or slide) of the graphical display presentation software and the inserted pen stroke data from the second pen may be displayed or displayable in a second document (or slide) of the graphical display presentation software. It is preferred that pen stroke data from the first and second pens can be inserted substantially simultaneously for display in the graphical display presentation software. This is made possible by both pens being wirelessly paired with the computing means at the same time, the plug-in being capable of selectively identifying the pen stroke data of the first and second pens, even if this data is received at substantially the same time.

The writing surface (or writing surfaces) may include a slide selection active area. This typically is subdivided into two or more active areas, each associated with a different slide, or each operable to cause the application to open a new slide for subsequent input of inserted pen stroke data.

The plug-in may include means for detecting a "create slide" command carried out by a user operating the pen on the writing surface in a pen stroke or series of pen strokes corresponding to a "create slide" command. This may be a pre-defined series of pen strokes carried out at any part of the writing surface. Alternatively, this may be at least one pen stroke carried out at a "create slide" region of the writing surface. The position-coding pattern at the "create slide" region of the writing surface detected during the pen stroke may then be interpreted by the plug-in as a "create slide" command to be passed on to the graphical display presentation software for creating a new slide in the graphical display presentation software.

Additionally or alternatively, the plug-in may include means for detecting a "create document" command carried out by a user operating the pen on the writing surface in a pen stroke or series of pen strokes corresponding to a "create document" command. This may be a pre-defined series of pen strokes carried out at any part of the writing surface.

Alternatively, this may be at least one pen stroke carried out at a "create document" region of the writing surface.

The position-coding pattern at the "create document" region of the writing surface detected during the pen stroke may then be interpreted by the plug-in as a "create document" command to be passed on to the graphical display presentation software for creating a new document in the graphical display presentation software.

The plug-in may include means for detecting a "same slide" command. The "same slide" command may be provided by the pen in a similar way to the "new slide" or "new document" commands discussed above. The "same slide" command is of use where there are multiple pens paired with the computing means and it is intended that the pen stroke data of each pen should be displayed together on the same slide.

Preferred embodiments of the invention will now be set out by way of example, with reference to the accompanying drawing, in which Fig. 1 shows an example layout of a suitable writing surface for use with an embodiment of the invention.

The embodiments of the present invention are intended for use with known digital pens and their accessories. One suitable example is the Logitech (registered trade mark) io2 (trade mark) Digital Writing System. Furthermore, the embodiments of the present invention are intended for use with known position-coding writing surfaces, particularly Anoto (registered trade mark) patterned paper. A suitable position-coding pattern for use with the present invention is described in detail in WO 2006/049573, the content of which is incorporated herein by reference in its entirety.

Examples of suitable patterns are disclosed in US 6,570,104, US 6,663,008 and US 6,667,695, which are also incorporated herein by reference in their entirety.

It is preferred that the digital pen for use with the present invention has wireless connectivity and in particular it is preferred that the digital pen can communicate via the Bluetooth wireless standard protocol referred to above.

The embodiment described below has particular relevance for interacting with the software application Microsoft PowerPoint (registered trade marks), and more specifically with Microsoft PowerPoint 2003 SP2, although the skilled person will see that the embodiments described below may be adapted to interact with another presentation graphics program.

The functionality of an exemplary embodiment will be described with reference to the operations of a user. The hardware elements of the present embodiment are an Anoto-enabled digital pen with short range wireless connectivity (e.g. Bluetooth), Anoto position coding paper and a personal computer having corresponding short range wireless connectivity (e.g. Bluetooth) . The computer may be connected to an auxiliary display such as a projector or other large-format display. This is because the embodiment has particular relevance to situations in which what is written or drawn on the paper using the pen is to be streamed in real time to the computer so that others can see the corresponding information on the large-format display also in real time.

The Anoto position coding paper has a pattern arrangement that instructs the pen to operate in Bluetooth mode. Such patterns are known already for use on pre-designed forms for use with mobile telephones, so that a form can be filled in using a digital pen, the data stored on the pen until a box on the paper is marked using the pen. The box contains the required send" instruction coded in the position-coding so that the pen sends the stored data to the mobile telephone which in turn sends the data to a central server so that the data can be input in the required format into a suitable database. In this way, neither the user nor any other person need input the data into the database from the form.

Further details are available from http://www.destinywireless.co.uk/pentech.aspx (accessed S January 2007) . Thus, Anoto functionality already allows for the pen to be instructed by the paper to operate in Bluetooth mode. In the present embodiment, the pattern on the paper instructs the pen to operate continuously in Bluetooth mode, so that the pen stroke data collected by the pen is streamed in real time to a paired personal computer.

The pen operates to image the pattern of dots on the paper writing surface when the pen detects a pen stroke event.

This is detected by a force sensor associated with a writing tip of the pen. Typically the writing tip is provided by a replaceable writing cartridge and the force sensor is coupled to the cartridge to sense force on the cartridge due to pressure between the writing tip and the writing surface.

In a known fashion, an imaging system of the pen receives a series of images of the writing surface. Each image corresponds to a unique position on the Anoto pattern, allowing its position to be uniquely determined and to generate a data series corresponding to pen stroke data for transmission from the pen. This is transmitted from the pen via a Bluetooth transceiver built into the pen to a paired personal computer. Suitable Bluetooth-enabled personal computers will be well known to the skilled person.

The computer may operate on a standard Microsoft Windows (registered trade marks) operating system such as Microsoft Windows XP. In the preferred embodiment, the computer runs Microsoft PowerPoint 2003 SP2 and an associated plug-in that will be described in further detail below.

The plug-in operates to allow pen input into slide creation and annotation in PowerPoint. The plug-in is a multi-threaded plug-in application which receives the pen stroke data, processes the pen stroke data and passes it on to PowerPoint for drawing into the PowerPoint slide.

The pen stroke data is received at the computer via Bluetooth and the Bluetooth stack is parsed to receive the data in its native Anoto format.

The pen strokes are interpreted using standard Anoto software in the same way as if the pen strokes were received via a digital pen dock after storage in the pen and subsequent docking of the pen in the dock.

There are two possible routes for further processing of the data. Either the plug-in passes the data on to PowerPoint for free-form writing and/or drawing, or the data can be further processed for conversion to ASCII text.

Conversion of pen stroke data to ASCII text is achieved using handwriting recognition (conversion) libraries.

Suitable software is provided by MyScript (registered trade mark) or similar from Vision Objects -see http://www.visionobjects.com (accessed 8 January 2007).

Such software is well known to the skilled person. The libraries pass back the best-fit converted ASCII text corresponding to the raw pen stroke data.

The plug-in provides text-size matching means so that the converted ASCII text can be inserted into the application slide at a font size that scales with the size of the characters written or drawn by the user.

In each case (i.e. where the graphics to be displayed in PowerPoint are either ASCII text or handwriting or drawing) the plug-in provides further processing of the data. This is processing to interpret and map the data onto PowerPoint including positional data, colour of pen used, thickness of line used, group data and free draw or snap draw (straight line drawing) . This processed data is then inserted into PowerPoint in the same way as data inserted into PowerPoint using a keyboard or mouse.

Using additional functionality, suitable operation of the pen on the paper (e.g. by executing a predetermined set of pen strokes, or by activating a predetermined functional box on the Anoto paper) allows the user to control which slide has data input from the pen. Furthermore, in a development of the present embodiment, a single computer can be paired with two or more pens so that data from both pens can be added into PowerPoint so that the data appears as graphics on a the same slide or on different slides substantially simultaneously. This functionality can be determined by appropriate selection of a functional box on the paper, or by executing a predetermined set of pen strokes.

The plug-in is coded using.Net/C#. This is an object-oriented programming language developed by Microsoft intended for writing applications for both hosted and embedded systems. The skilled person has no difficulty in preparing suitable source code via known methods in order to code the plug-in.

Fig. 1 shows the layout of an exemplary writing surface for use with an embodiment of the present invention. Note that this drawing does not show the specific Anoto position-coding pattern to be used with the embodiment of the invention, but shows the areas in which such position-coding patterns will be placed.

First active area 10 occupies about 80% of the area of the writing surface. First active area 10 has a generally rectangular shape, substantially corresponding to an aspect ratio of a slide in landscape format in Microsoft PowerPoint.

The first active area has a central marker 11 for indicating the centre of the first active area and also has gridlines 32 for guidance to the user during drawing or writing operations.

Although not shown in Fig. 1, first active area 10 includes a position-coding pattern that instructs the digital pen to operate in Bluetooth real time streaming mode. In use, a digital pen pairs via Bluetooth with an associated computer.

Subsequently, touching the writing tip of the pen onto the first active area causes the pen to image the coding pattern of the first active area, which in turn causes the pen to stream pen stroke data to the computer. The plug-in receives this data in the way discussed above. This causes a registration event for the pen. In the preferred embodiment, this causes a new slide to be opened in the PowerPoint application, and the pen stroke data is inserted into this slide as discussed above.

Outside first active area 10 are provided further active areas. Area 12 is a slide association tool. It is intended that this area provides a choice of six slides for the user to input pen stroke data into. For example, a user may decide that subsequent pen stroke data should be input not into the first slide but into a different slide. In this case the user touches the writing tip of the pen into, for example, box 2 of slide association tool 12. If a corresponding slide is not yet open, this action causes a new slide (second slide) to open. Subsequent pen stroke data in the first active area is input into the second slide.

If a corresponding second slide is already open, subsequent pen stroke data is inserted into the second slide.

As can be seen from Fig. 1, the present embodiment supports the opening of six such slides in this way, but the skilled person will see that it is straightforward for a different number of slides to be operated in this way, e.g. ten or more.

It is possible for the embodiment to operate so that a multiple number of digital pens is used with the system, e.g. one pen per user. For example, each pen (and user) may have an associated writing surface such as illustrated in Fig. 1.

In that case, it may be appropriate for each user to select a different one of the boxes shown in slide association tool 12, so that each user inputs pen stroke data into different slides in the PowerPoint application.

Still further active areas are shown to the right of first active area 10. Active area (pidget) 14 corresponds to a "black" command to the plug-in, to indicate that the subsequent pen strokes in active area 10 are to be represented on the displayed slide in black. Pidgets 16, 18 and 20 correspond to "red", "blue" and "green" commands in a similar way. These pidgets are activated by the user touching or drawing in the active area of the pidget. This allows the pen to image the corresponding coding pattern in the pidget and stream this information to the computer.

Pidgets 22, 24 and 26 correspond to a handwriting conversion tool, free form drawing tool and straightline ("snap") drawing tool, respectively, and these operate in a similar way to the pidgets discussed above.

In a modified embodiment, it may be preferred to have only a single active area (corresponding to first active area 10) that extends substantially across the whole writing surface.

In this case, there are not provided further active areas.

This has an advantage of simplicity.

In a further modified embodiment, it may be preferred that the first active area has content (other than the position-coding pattern) pre-printed. The use of this embodiment may be that the user or users may add to the pre-printed content.

The pre-printed content may also be shown on the slide of the application into which the pen stroke data is to be inserted. This may be of use in a teaching exercise, and the pre-printed content may be a partially completed table, graph or similar.

The above embodiments are described by way of example.

Modifications of these embodiments, further embodiments and modifications thereof will be apparent to the skilled person on reading this disclosure and as such are within the scope of the invention.

Claims

CLAIMS1. A method of operating a digital pen interaction system, the system including: a computing means; a digital pen; and a position-coding writing surface, wherein: the digital pen is operated to generate pen stroke data corresponding to positions of a writing tip of the pen on the position-coding writing surface when the digital pen senses a writing event, and said pen sends said pen stroke data wirelessly substantially in real time to said computing means; the computing means has at least one associated display means and receives said pen stroke data wirelessly substantially in real time from said digital pen and processes said pen stroke data to be displayed on said display means via a graphical display software application substantially in real time.
2. A method according to claim 1 wherein the writing surface is a markable writing surface.
3. A method according to claim 1 or claim 2 wherein the writing surface includes a first active area that occupies a major portion of the writing surface.
4. A method according to claim 3 wherein the position-coding of the first active area, when sensed by the digital pen, instructs the digital pen to operate in wireless streaming mode.
5. A method according to claim 3 or claim 4 wherein at least part of the perimeter of the first active area is marked so that the user can identify the perimeter of the first active area.
6. A method according to any one of claims 1 to 5 wherein the digital pen includes an ink reservoir.
7. A method according to any one of claims 1 to 6 wherein the digital pen and computing means operate according to wireless communications protocol IEEE 802.15.1.
8. A method according to any one of claims 1 to 7 wherein the graphical display software application is suitable for the preparation of electronic display slides.
9. A method according to claim 8 wherein the graphical display software application has a "view slide show" mode and the inserted pen stroke data is insertable into the slide or slides of the application for display during "view slide show" mode.
10. A method according to any one of claims 1 to 9 wherein there is provided a plug-in application for interfacing with the graphical display software application to provide a data receiving and processing interface between a wireless communications stack of the computing means and the graphical display software application.
11. A method according to claim 10 wherein the plug-in application provides means for passing the interpreted pen stroke data to handwriting conversion means for conversion of handwritten characters or symbols into ASCII characters and/or typical word processing functions.
12. A method according to claim 11 wherein the plug-in application is provided with text size matching means for selecting an appropriate font size for ASCII characters to scale with a user-chosen size of the handwritten characters or symbols.
13. A method according to any one of claims 10 to 12 wherein the plug-in application provides further interpretation of the pen stroke data in order for at least one further characteristic to be determined, for application of a corresponding characteristic to the inserted pen stroke data displayed by the graphical presentation display application.
14. A method according to claim 13 wherein said at least one characteristic is selected from colour, line thickness, grouping, free draw and snap draw.
15. A method according to claim 13 or claim 14 including the step of the user selectively activating one or more active functional areas of the writing surface.
16. A method according to claim 15 wherein the writing surface includes a slide selection active area operable to cause the application to open a new slide for subsequent input of inserted pen stroke data.
17. A method according to any one of claims 1 to 16 in which a second digital pen is provided, similar to the first digital pen, and is operated by a second user, and the second digital pen interacts with the graphical display presentation application at the same time as the first digital pen.
18. A method according to claim 17 wherein the computing means is operable to differentiate between pen stroke data from the first and second digital pens.
19. A method according to claim 18 in which there is provided a second writing surface for interaction with the second digital pen.
20. Computing means operatively programmed to carry out the method according to any one of claims 1 to 19.
21. A computer program capable of carrying out the method according to any one of claims 1 to 19 when loaded onto a suitable computing means.
22. A storage medium such as an optical or magnetic disk, or a solid state storage medium, storing the computer program according to claim 21.
23. A kit including: a computer program according to claim 21 or a storage medium according to claim 22; a position-coding writing surface; and a digital pen capable of generating pen stroke data corresponding to positions of a writing tip of the pen on the position-coding writing surface when the digital pen senses a writing event, and capable of sending said pen stroke data wirelessly substantially in real time to a corresponding computing means.
24. A digital pen interaction system, the system including: a computing means; a digital pen; and a position-coding writing surface, wherein: the digital pen is operable to generate pen stroke data corresponding to positions of a writing tip of the pen on the position-coding writing surface when the digital pen senses a writing event, and operable to send said pen stroke data wirelessly substantially in real time to said computing means; the computing means has at least one associated display means and said computing means is operable to receive said pen stroke data wirelessly substantially in real time from said digital pen and to process said pen stroke data to be displayed on said display means via a graphical display software application substantially in real time.