GB2425698A

GB2425698A - Position encoding patterns, and systems and method using the same

Info

Publication number: GB2425698A
Application number: GB0508168A
Authority: GB
Inventors: Andreu Gonzalez
Original assignee: Hewlett Packard Development Co LP
Current assignee: Hewlett Packard Development Co LP
Priority date: 2005-04-25
Filing date: 2005-04-25
Publication date: 2006-11-01
Also published as: GB0508168D0

Abstract

A substrate carries a background pattern, in which local portions of the background pattern uniquely identify a location of the background pattern over the substrate. The local portions identify a location according to predetermined interpretation criteria. The background pattern comprises a plurality of regions, each region comprising at least one local portion, wherein the regions are arranged in positions over the substrate such that, for a plurality of the regions, the location or locations identified by the local portion or portions of the region, according to the interpretation criteria, do not correspond with the position of the local portion or portions over the substrate. The substrate thus has re-arranged portions, so that the location information received, when the substrate is used in a digital pen and paper system, does not provide the correct location information. The specific arrangement of the regions over the substrate is required to enable the location information to be correctly interpreted.

Description

I

POSITION ENCODING PATTERNS, AND SYSTEMS AND METHOD USING THE

SAME

Field of the invention

This invention relates to position encoding substrates, which translate the position of written text and images on the substrate into digital data. In particular, the invention relates to the use of a substrate (such as paper) with a position encoding pattern, which is used in conjunction with a so-called "digital pen". This pen reads the local pattern when the pen is in use, in order to derive position (or position- dependent) information. This technology is generally known as "digital pen and paper" technology.

Background of the invention

The paper used in digital pen and paper technology carries a non-uniform background pattern of dots. The pattern of dots at any location encodes location information for the location on the page. An overlying image is typically provided as a watermark so that the dots of the background pattern can be distinguished. The local dot pattern on all locations of the background pattern is unique and identifies the location. The number of different possible local dot patterns is preferably sufficiently large that a large number of different pages can be provided with different background patterns, with different local dot patterns at all locations for all pages.

Thus, any local dot pattern can identify not only a position on the page, but can also identify a page of a multiple page document.

To use a digital pen and paper system, a user has an electronic pen for writing over the background pattern. This pen is both a reading and a writing device, and allows the user to write over the paper where he/she chooses, but also identifies the locations at which markings are written by reading the local background pattern. The pen typically comprises a digital (infra-red) camera, an image processing unit and a wireless transceiver, and is battery operated. The pen also writes in ink, so that from the user's point of view, the pen has normal functionality. The ink from the pen is not visible to the infra-red camera.

There are many possible uses for digital pen and paper technology.

Essentially, it provides a means of digitising handwritten text and drawings at the same time that the information is being written on the paper. This provides immediate digital information for further handling and processing. There are many more specific applications of the technology. For example, the technology can be used for creating a data file providing the information which has been handwritten on a paper form.

There are many different possible encoding schemes which can be used for the background pattern. However, the pattern typically comprises a large number of small dots printed on the paper in an asymmetrical pattern. By way of example, the dots in the pattern may have a nominal spacing of around 0.3mm. The pattern is arranged such that the dots are displaced from a strict grid arrangement, and the local dot pattern detected by the camera can then be used to determine the pen position.

A printed document will use a predetermined encoding scheme to encode the positions of the background pattern. Thus, an isolated system can determine position information from information read by the digital pen, even when this system has not been used in the printing of the document. This gives the system a universal applicability, but it also means that the data received from the digital pen is not secure, as it encodes the written information in a standard, predetermined manner.

Summary of the invention

According to the invention, there is provided a substrate carrying a background pattern, in which local portions of the background pattern provide unique encoding information dependent on the respective location of the local portion of the background pattern over the substrate, wherein the local portions provide encoding information according to predetermined interpretation criteria, wherein the background pattern comprises a plurality of regions, each region comprising at least one local portion, wherein the regions are arranged in positions over the substrate such that, for a plurality of the regions, the encoded information identified by the local portion or portions of the region, according to the interpretation criteria, does not correspond with the encoded information corresponding to the position of the local portion or portions over the substrate.

The substrate has re-arranged portions, so that the location information received when the substrate is used in a digital pen and paper system, does not provide the correct location information. The specific rearrangement of the regions over the substrate is required to enable the location information to be correctly interpreted.

The substrate can be used in a system which further comprises a reading and writing device for reading a local background pattern and writing on the substrate, and a means for interpreting the local portions of the background pattern and determining a corresponding position using the predetermined interpretation criteria.

The invention also provides a method of producing an image including a background pattern, in which local portions of the background pattern provide unique encoding information dependent on the respective location of the local portion of the background pattern over the substrate, wherein the local portions provide encoding information according to predetermined interpretation criteria, the method comprising: dividing a starting background pattern into regions, each region comprising at least one local portion, wherein the positions of the local portions of the starting background pattern provide encoded information in accordance with the predetermined interpretation criteria; re-arranging the regions to define a different combination of positions over the substrate thereby providing a re-arranged background pattern, providing the re-arranged background pattern on a substrate.

The re-arrangement of regions provided by this method provides security for the information received when the image is used in a digital pen and paper system.

The invention also provides a method of obtaining position information from data received from an input device used in conjunction with an image having a

background pattern, comprising:

producing an image using the method of the invention; stonng data concerning the re-arranging step; receiving encoded information from local portions of the re-arranged

background pattern; and

interpreting the encoded information using the data concerning the rearranging step and the predetermined interpretation criteria.

The invention also provides a computer program for controlling a printer to print an image including the background pattern, and a computer program for obtaining position information from data received from an input device used in conjunction with an image having the background pattern.

Brief description of the drawings

Examples of the invention will be described in detail with reference to the accompanying drawing, in which: Figure 1 shows the paper used in digital pen and paper technology; Figure 2 shows an enlarged part of the background pattern of the paper of Figure 1; Figure 3 shows the pen used in digital pen and paper technology; Figure 4 is used to explain one way in which a dot pattern can encode position information; Figure 5 shows an example of system of the invention; Figure 6 shows how the pattern rearranging is performed in one example of the system of the invention; and Figures 7a to 7c show the effect of the pattern re-arranging.

Detailed description

Figure 1 shows in simplified manner a form which may be printed for use with a digital pen of a system using digital pen and paper technology. This is one example of possible application of digital pen and paper technology. There are many other applications, and the technology can be seen as a system enabling user-friendly and real-time digitisation of hand-written information.

The form 10 has text and images 12 and spaces 14 where the user is required to write to complete the form.

The form is provided on a substrate carrying, in this example, a nonuniform background pattern of dots. The pattern of dots at any location encodes location information for the location on the page. The form is provided as a watermark so that the dots of the background pattern can still be distinguished.

The local dot pattern on all locations of the background pattern is unique and identifies the location. Furthermore, a local dot pattern for multiple pages can be different. Thus, any local dot pattern can identify not only the position on the page, but also the page out of a document comprising multiple pages.

To use this type of system, the user has an electronic pen for writing over the form. This pen is both a reading and a writing device, and allows the user to write over the form where he/she chooses but also identifies the locations at which markings are written by reading the local background pattern. The pen stores in a memory the dot patterns at the locations of all written markings.

This provides the user with an easy interface, where the user is simply required to write over a form in conventional manner. The system recognises the location of marks made on the form, and can of course perform character recognition for text entries.

The technology for defining the background dot pattern and for the electronic pen is already available. For this reason, only a brief discussion of the technology will now be given.

Figure 2 shows an enlarged part of the dot pattern. The pattern is slightly displaced from an orthogonal grid. In one example, an area of 6x6 dots gives a unique position. With each dot in one of four possible positions, this allows 4 (=272) different locations to be encoded. A 6x6 dot area covers less than 2mm x 2mm, and the overlap of 6x6 dot areas enables adjacent identifiable locations to be spaced only 0.3mm apart.

Figure 3 shows the digital pen and computer with which it communicates.

The pen 30 comprises a nib 32 coupled to an ink cartridge 34 and pressure sensor 36 for detecting when markings are being written. An image sensor camera 38 detects the local dot pattern. The pressure sensor and camera supply data to a processor 40. When information is being written, the pen detects this using the pressure sensor and at this time the image sensor, for example camera, detects the local dot pattern under the control of the processor 40.

The pen has a memory 42 for storing the camera information, either as recognised dot patterns or as the corresponding locations, of all written markings.

The pen does not require significant processing power, but simply requires a database of all recognisable dot patterns and a memory for storing sequences of dot patterns corresponding to the positions of the written markings. The pen is stand alone and is powered by battery 44. The electronic pen can therefore be produced at a relatively low cost.

The information stored in the electronic pen can be downloaded into a central computer 50 having a receiver 54 and processor 56, with which the pen communicates using transmitter 52.

The system is designed as a user interface which simulates conventional pen and paper writing for the user. The system has many additional functions, and these can be applied to this invention. For example, each page may include dedicated areas where character recognition is carried out. Other dedicated areas may cause the computer to which the information is downloaded to generate and send emails to addresses obtained by the character recognition. The ways in which these functions can be used will be apparent to those skilled in the art.

In one preferred encoding scheme, the dot may be displaced by a fixed amount in any one of the four axial directions (i.e. up, down, left or right). Each dot thus has four possible positions. A sub-array of dots then encodes one location. This sub-array may have a variety of sizes, depending on the resolution required and the number of locations to be encoded.

Figure 4 shows an example of this encoding scheme, in which each dot is displaced by a fixed amount from the reference grid (in hatched lines) in one of the four axial directions. The dot pattern also enables the reference grid position to be determined, in a manner which will be known to those familiar with digital pen and paper technology.

A portion of the pattern consisting of nxm markings may be assumed to represent the minimum amount of pattern necessary to decode a location, for example the 4x4 pattern of dots in Figure 4 may be the minimum size pattern, although larger size patterns, such as 6x6 as mentioned above, can be used to encode larger numbers of locations.

This invention concerns specifically the security of the data transmitted from the pen to the host computer. If the location information is encoded using a standard protocol for the interpretation of the local dot patterns, then the wireless information transmitted from the pen will provide all the information needed by a third party to derive the pen markings made by the user.

The examples of the invention described below address this issue by rearranging the background pattern before printing the background image.

In particular, the background pattern, or a given portion of the background pattern, is split into smaller regions, creating a puzzle of the original pattern. A disordered version of the initial pattern is printed, by placing pieces of the puzzle (i.e. different regions) in a different location with respect to the original pattern, while covenng the same area. Each region may be in a new location, or else the new locations may be randomly generated (in which case some regions could end up in the same place).

A re-arranging scheme must be used which has a large number of possible rearrangement results, so that the original pattern cannot easily be recovered by trial and error.

Writing with a digital pen on this disordered printed pattern will produce incoherent data, since a continuous pen input stroke will lead to a disarranged digitally captured output stroke, when interpreted using the standard protocol.

In this way, only authorized applications can be made able to decipher the received data and obtain the pen strokes in their original locations, for subsequent processing.

Figure 5 shows a system of the invention, for generation of the background image on a substrate and the pen reading system.

The system comprises the substrate 10 which carries the background image, as explained above. The pen 30 provides output data as a set of strokes 60, and these are output in the form of the local dot patterns over which the pen has been passed during application of pressure. This information is provided to the software application 62 associated with the particular substrate 10, which may for example be a form requiring completion.

The substrate with the background pattern can be printed using a conventional printer 64, and the printer is controlled to print the combination of the background pattern and another image, for example a digital pdf form, shown as 66, which is then associated with an adobe (Trade Mark) application 67.

The data stored in the pen 30 is not encrypted, so when sent to the application 62, this data is prone to be understood by any intruder software. Therefore the data processes that occur between the filling in of the form and the subsequent processing stages are insecure, and do not request any validation/authentication.

To the extent described above, the system of Figure 5 is conventional.

The system is improved by incorporating the mechanism of disordering the position identification pattern so that the written information is meaningless throughout all subsequent processes, and only authorized applications are able to interpret the information, for example by recovering a special key from a centralized component.

For this purpose, the system of Figure 5 has a system controller 68 which communicates with the application 62 and with a printer driver 70, and the controller is used to implement the re-arranging function. The rearranged background pattern is then provided to a standard printer driver 72.

The re-arranging function will now be described, with reference to Figure 6.

The background pattern 80 is divided into sub-regions 82, and these are re- arranged and relocated (into positions 82') 50 as to form a different pattern 80', while preserving other qualities such as the pattern area and structure, for example the rectangular shape in the example of Figure 6. The original pattern is on the left of Figure 6, and the local dot patterns are arranged in positions which are in accordance with the predetermined interpretation protocols.

The disordered pattern does not contain the encoded information in the same locations as before, and is therefore not in accordance with the predetermined interpretation protocols. Writing on the disordered printed pattern therefore produces incomprehensible data for the predetermined interpretation criteria, unless the original order can be reproduced.

In the example of Figure 6, the pattern area 80 covers most of the page, and is divided into 165 regions (11 horizontal divisions, and 15 vertical divisions). For explanation, each region is given a label, using a letter for the horizontal position and a number for the vertical position. Figure 6 illustrates the example of region "F12" being moved to position "Hi 1", and region "14" being moved to position "C8".

There is an enormous number of ways of relocating each region so that it generates a different pattern.

Preferably, the disordered versions of the pattern preserves all the geometric pattern properties, such as the area and the outline, so that there is no discrimination between the intended pattern layout and the rearranged pattern.

This enables the relationship between the overlying image regions and regions where location identification is provided, to be preserved. If a Digital Form is created, consisting of form contents and pattern areas underneath, the pattern areas will be in chosen areas where the form is to be completed. Relationships between the form content and pattern should thus be preserved so that a digital pen can capture written information in the same target areas, and the system is subsequently able to treat captured data appropriately, according to its contextual meaning, such as ticks on check boxes or handwriting on character recognition fields.

In order to keep the geometric properties of the pattern layout, it is preferable to divide the pattern into regions of equal size and shape. In this way, each region can be fitted on a region previously occupied by another region. Of course, the regions do not necessarily need to be square or even rectangular.

Each pattern region contains at least enough of the background pattern so that the imaging device is able to decode the intended information contained in the pattern. For example, the minimum pattern unit may be a 4x4 or 6x6 dot sub-array.

The resulting disordered pattern has the property that the regions are arranged in positions over the substrate such that the location or locations identified by the dot patterns in those regions do not correspond with the actual position over the substrate, according to the interpretation criteria. This is preferably the case for at least half of the regions, and more preferably substantially all of the regions are moved into new positions. However a substantially random rearrangement could of course leave some regions in the same place. Preferably at least 80% and more preferably at least 90% of the regions are in new locations.

With reference to Figure 5, the use of the system will be explained.

The image of the digital form 66 is combined with data identifying the desired background pattern, which satisfies the rules of the interpretation protocol. The adobe (Trade Mark) application 67 introduces this background pattern data.

The digital paper printer driver 70 accesses the background pattern data embedded in the Digital Form, which includes at least a reference coordinate xO,yO, and the width and height of the pattern area.

The digital paper printer driver 70 requests the image for the background pattern from the system controller 68, which provides a description of the (original) pattern.

The digital paper printer driver 70 performs the pattern division into regions and generates the disordered version for printing.

The digital paper printer driver 70 also sends to the system controller 68 a key' describing the rules used to generate the disordered pattern.

The user fills the printed form 10, and the pen generates location information which, according to the standard interpretation protocol, does not correspond to the actual position with respect to the page. The captured data strokes are therefore encrypted.

During processing of the captured pen data, the application 62 receives an initially unintelligible set of coordinates. In order to recover the actual data written by the user, the application 62 communicates with the system controller 68 to retrieve the key' describing the correct order.

The pattern regions are selected to be small enough that the disordered pattern cannot be decrypted by image analysis or other techniques. This requires the division to be sufficiently small that a large number of permutations exist.

Figure 7 illustrates how the data strokes will actually be captured when disordered patterns are used.

Figure 7a shows the original text as written by the user. Figure 7b shows the text divided into the regions, and Figure 7c shows the effect of disordering the pattern on the captured pen strokes.

The system adds security on the user filling stage, so that data stored in the pen is intentionally codified as a set of incoherent data point (or strokes). The system is transparent to the end user, in that the user writes on the digital paper in normal manner. The system can be implemented with the standard hardware of the digital pen and paper system. There is no need to generate new patterns or interpretation protocols, and the existing standard protocols can be used.

A full page of the pattern used on Digital Pen and Paper can be easily divided into more than 1900 partitions (for example 50 x 40, corresponding to unit sizes of approximately 6mm x 5mm for an A4 page), giving a large number of permutations of the order of 1900! (= 101073646) different ways of rearranging the pattern.

The background pattern may occupy all or substantially all of the area of the printed substrate. However, it may instead occupy only selected areas of the substrate, where markings are to be made. In many applications, there may only be predetermined sub-areas where the user is expected to provide markings, and the substrate design will take into account the desired use of the substrate.

The encryption provided by the re-arranging of regions as described above may be applied to all of the background pattern, but it may instead only be applied to selected areas, for example areas where sensitive information is to be written. Other parts of the pattern may be left unencrypted.

Only one design of dot pattern has been explained above. This is however only one example for the purposes of explanation. Other designs of dot pattern or indeed pattern of lines can be used to encode local information. When using a dot pattern, the smallest size used to encode information can be square or another shape, and the size can be selected according to the desired resolution. For example, the minimum size may be a square of 4x4, 5x5, 6x6 etc., or rectangular, for

example 10x2, etc.

In the example above, the local portions of the background pattern uniquely identify the respective location of the background pattern over the substrate (according to the predetermined interpretation criteria). Thus, the local dot (or line) patterns are interpreted by the criteria to derive a position co-ordinate. This position information can then be interpreted further using information concerning the printed image, as this dictates the meaning of markings made. The pattern may instead provide other unique encoding information, again dependent on the respective location of the background pattern over the substrate. The information is location- dependent, in that the position of the pen over the page needs to be known in order to know that a particular location has been selected by the user. However, the information encoded may provide direct information from the location.

For example, the local pattern may encode letters of the alphabet which correspond to the location over the substrate (for a touch sensitive keyboard) without the need to actually encode position information.

As another example, the local pattern may encode a command to do something, for example a print command, and the local pattern can then be provided over a print icon. Other parameters, such as time, may be encoded by the

background pattern.

Various other possibilities will be apparent to those skilled in the art.

Claims

Claims 1. A substrate carrying a background pattern, in which local

portions of the background pattern provide unique encoding information dependent on the respective location of the local portion of the background pattern over the substrate, wherein the local portions provide encoding information according to predetermined interpretation criteria, wherein the background pattern comprises a plurality of regions, each region comprising at least one local portion, wherein the regions are arranged in positions over the substrate such that, for a plurality of the regions, the encoded information identified by the local portion or portions of the region, according to the interpretation criteria, does not correspond with the encoded information corresponding to the position of the local portion or portions over the substrate.
2. A substrate as claimed in claim 1, wherein the regions comprise rectangular regions.
3. A substrate as claimed in claim I or 2, wherein the regions are arranged in positions such that the locations identified corresponding to an image on the substrate correspond to a disordered version of the image on the substrate.
4. A substrate as claimed in claim 3, wherein the size of each region is selected such that the disordered version comprises an encrypted version of the image on the substrate.
5. A substrate as claimed in any preceding claim, wherein the background pattern comprises a pattern of dots, and wherein each local portion of the pattern comprises a sub-pattern of dots.
6. A system for encoding position on a substrate, comprising a substrate carrying a background pattern, in which local portions of the background pattern provide unique encoding information dependent on the respective location of the local portion of the background pattern over the substrate, a reading and writing device for reading a local background pattern and writing on the substrate, and a means for interpreting the local portions of the background pattern and determining the corresponding encoded information using predetermined interpretation criteria, wherein the background pattern comprises a plurality of regions, each region comprising at least one local portion, wherein the regions are arranged in positions over the substrate such that, for a plurality of the regions, the encoded information identified by the local portion or portions of the region, according to the interpretation criteria, does not correspond with the encoded information corresponding to the position of the local portion or portions over the substrate.
7. A system as claimed in claim 6, wherein the reading and writing device comprises an image capture device, an image processing means and a transmitter.
8. A substrate as claimed in claim 6 or 7, wherein the regions are arranged in positions such that the locations identified corresponding to an image on the substrate correspond to a disordered version of the image on the substrate.
9. A substrate as claimed in claim 8, wherein the size of each region is selected such that the disordered version comprises an encrypted version of the image on the substrate.
10. A system as claimed in any one of claims 6 to 9, further comprising a system controller, adapted to: record data concerning the arrangement of regions for a given printed substrate; and interpret location information received from the reading and writing device from the given printed substrate using the data concerning the arrangement of regions.
11. A method of producing an image including a background pattern, in which local portions of the background pattern provide unique encoding information dependent on the respective location of the local portion of the background pattern over the substrate, wherein the local portions provide encoding information according to predetermined interpretation criteria, the method comprising: dividing a starting background pattern into regions, each region comprising at least one local portion, wherein the positions of the local portions of the starting background pattern provide encoded information in accordance with the predetermined interpretation criteria; re-arranging the regions to define a different combination of positions over the substrate thereby providing a rearranged background pattern, providing the re-arranged background pattern on a substrate.
12. A method as claimed in claim 11, wherein the re-arranging is such that, for a plurality of the regions, the encoded information identified by the local portion or portions of the region, according to the interpretation criteria, do not correspond with the encoded information corresponding to the position of the local portion or portions over the substrate.
13. A method as claimed in claim 11 or 12, wherein the providing step comprises printing.
14. A method of obtaining position information from data received from an input device used in conjunction with an image having a background pattern, comprising: producing an image using the method as claimed in claim 11; stonng data concerning the re-arranging step; receiving encoded information from local portions of the re-arranged

background pattern; and

interpreting the encoded information using the data concerning the rearranging step and the predetermined interpretation criteria.
15. A computer program for controlling a printer to print an image including a background pattern, in which local portions of the background pattern provide unique encoding information dependent on the respective location of the local portion of the background pattern over the print medium, wherein the local portions provide encoding information according to predetermined interpretation criteria, wherein the computer program is adapted to: divide a starting background pattern into regions, each region comprising at least one local portion, wherein the positions of the local portions of the starting background pattern provide unique encoding information dependent on the respective location of the local portion of the background pattern over the substrate according to predetermined interpretation criteria, re-arrange the regions to define a different combination of positions over the substrate thereby providing a re-arranged background pattern; and print the re-arranged background pattern on a substrate.
16. A computer program as claimed in claim 15, wherein the computer program is adapted to re-arrange the regions such that, for a plurality of the regions, the encoded information identified by the local portion or portions of the region, according to the interpretation criteria, does not correspond with the encoded information corresponding to the position of the local portion or portions over the substrate.
17. A computer program for obtaining position information from data received from an input device used in conjunction with an image having a background pattern, in which local portions of the background pattern provide unique encoding information dependent on the respective location of the local portion of the background pattern over the print medium, wherein the local portions provide encoding information according to predetermined interpretation criteria, the computer program comprising: a first computer program portion for controlling a printer, as claimed in claim 15; a second computer program portion, adapted to: store data concerning the re-arranging step performed in the first computer program portion; receive information concerning local portions of the re-arranged background pattern; and interpret the local portion information using the data concerning the re- arranging step and the predetermined interpretation criteria.