GB2419664A

GB2419664A - Digital pen

Info

Publication number: GB2419664A
Application number: GB0424057A
Authority: GB
Inventors: Andrew Mackenzie; Victor Lara
Original assignee: Hewlett Packard Development Co LP
Current assignee: Hewlett Packard Development Co LP
Priority date: 2004-10-29
Filing date: 2004-10-29
Publication date: 2006-05-03
Also published as: GB0424057D0

Abstract

A digital pen 8 comprises a nib 10 arranged to be moved over a document 2. The pen 8 also comprises an imaging system 12 arranged to image an area of data encoding pattern 6 on the document 2. The pen 8 further comprises calibration means arranged to determine the position and orientation of the imaging system 12 relative to the nib 10.

Description

DIGITAL PEN

FIELD OF THE INVENTION

The present invention relates to digital pens.

BACKGROUND TO THE INVENTION

It is known to provide data encoding pattern on products, for example documents, so that a hand held device, such as a pen, can read the data encoded in the pattern and use it, for example, to detect its position as it is moved over the document. Where the hand held device is a pen arranged to mark the product, the position of marks made on the documents can be determined by the pen, thereby enabling the position of the marks on the document to be stored electronically as they are made.

Such pens comprise an imaging system having a lens, a lens holder or barrel and an image sensor for imaging the pattern and a nib for marking the product and the position of the nib relative to components of the imaging system must be known accurately in order for the position of the marks to be determined effectively. When the pen is manufactured it is not always possible to position the camera exactly relative to the nib.

Also during the lifetime of the pen it is possible that the nib can become mechanically misaligned with the components of the imaging system. The image formed on the sensor (and hence the calculated position of the nib) can be affected by movements or expansion/contraction in many parts of the imaging system: e.g. the assembly holding the nib, the parts between the nib assembly and sensor assembly, lens, lens holder, sensor mounted on barrel, parts mounting lens assembly, etc. * S *** * * S S * : : : : ** : : *S S S * * S * * S *S ** S *** S These movements of the camera relative to the nib can be vertical (parallel to the axis of the nib), horizontal (lateral movements perpendicular to the axis of the nib), or rotational movements of the sensor. These movements can lead to incorrect determination of the position of an imaged pattern or mark. In the case of data processing systems that interpret mark positions (e.g. if a box is selected with a tick' mark etc.) this error can lead to data interpretation errors or data loss, as the digital representation of the mark is not aligned with the known background in the same manner as the physical mark made by the user.

SUMMARY OF THE INVENTION

According to a first aspect of the invention, there is provided a digital pen comprising a nib arranged to be moved over a document, an imaging system arranged to image an area of data encoding pattern on the document and calibration means arranged to determine the position and orientation of the imaging system relative to the nib.

According to a further aspect of the invention, there is provided a method of calibrating a digital pen according to the invention comprising the steps of: locating the nib in a known position and orientation relative to a calibration target; imaging a test area comprising at least part of the calibration target; and determining the position and orientation of the nib relative to the imaging system from the test area image.

S S

S * * * * S S S S S *S* S * S S * S S * * *5 * 5 * S S * * * * S S ** S. * S.. S *

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the present invention will now be described by way of example only with reference to the accompanying drawings, in which: Figure 1 shows schematically a document and a digital pen according to an embodiment of the invention for use with the document; Figure 2 is a schematic representation of a pen cap having calibration means for use with the pen of Figure 1; Figure 3 is a schematic representation of a pen cap having calibration means according to a further embodiment of the invention; Figure 4 is a schematic diagram of a method of calibrating a digital pen; Figure 5 is a schematic diagram of an alternative method to that shown in Figure 4 for calibrating a digital pen; Figure 6 shows in detail part of a position identifying pattern used in the cap of Figure 3; Figure 7 is a schematic representation of part of calibration means according to a further embodiment of the invention including the pattern of Figure 6;

S S

* S *S* * * * * * S * * * C S I S * I I * S. * I *. S * * * . S * S C * *. S. * *S* * * Figure 8 is a schematic representation of part of calibration means according to a further embodiment of the invention; and Figure 9 is a schematic diagram of a method of calibrating a digital pen including the calibration means shown in Figures 7 and 8.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to Figure 1, a document 2 for use in a digital pen and paper system comprises a carrier 3 in the form of a single sheet of paper 4 with position identifying markings 5 printed on some parts of it. The markings 5, which are not shown to scale in Figure 1, form a position identifying pattern 6 on the document 2. Also printed on the paper 4 are further markings 7 which are clearly visible to a human user of the document, and which make up the human visible content of the document 2. The content 7 is in the form of a number of lines and text and graphic features which extend over, and are therefore superimposed upon, the pattern 6.

The pen 8 comprises a writing nib 10, and a camera 12 made up of a lens, a lens holder, an infra red (IR) LED 14 and a CMOS sensor 16. The camera 12 is arranged to image a circular area adjacent to the tip 11 of the pen nib 10. A processor 18 processes images from the camera 12 taken at a predetermined rapid sample rate. A pressure sensor 20 detects when the nib 10 is in contact with the document 2 and triggers operation of the camera 12. Whenever the pen is being used on an area of the document 2 having the pattern 6 on it, the processor 18 can determine from the pattern 6 the position of the nib 10 of the pen whenever it is in contact with the document 2. For this calculation it first determines the position of the camera from the pattern imaged by the camera, and then uses a nominal offset vector (distance and direction) from the nib to the imaging system that is either calculated at design time or measured during S..e * . . :: pen manufacture. From this it can determine the position and shape of any marks made on the patterned areas of the document 2. This information is stored in a memory 22 in the pen as it is being used. When the user has finished marking the document 2, this is recorded in a document completion process, for example by making a mark with the pen 8 in a send box 9. The pen is arranged to recognise the pattern in the send box 9 and send the pen stroke data to a pen stroke interpretation system in a suitable manner, for example via a radio transceiver 24 which provides a Bluetooth radio link with an internet connected PC, or via a USB docking station. Suitable pens are available from Logitech under the trade mark Logitech Jo, and from Nokia.

The pen 8 further comprises a pen cap 100 including a calibration mechanism according to a first embodiment of the invention, shown in Figure 2. The cap 100 comprises a casing 102 of generally circular cross section along its length. The size and diameter of the cap 100 is arranged such that it is a sliding fit over the end of the pen 18. A circumferential ridge 104 is provided on an interior wall 106 of the cap that cooperates with a circumferential groove 107 in the pen to allow snap fitting of the cap 100 over the end of the pen 8 to cover the nib 10 and camera 12 in a standard manner. In addition, a longitudinal ridge 108 is provided running generally between a closed end 110 of the cap 100 and an opposite open end 112. The ridge 108 is locatable within a corresponding longitudinal groove 113 formed in an exterior wall of the pen 8 (see Figure 1) such that when the cap 100 is placed on the pen 8, the nib 10 is aligned in exactly the same orientation relative to the cap 100 every time that it is snap fitted into place.

Towards the closed end 110 of the cap 100, there is located a calibration target in the form of a patch of data encoding pattern 114 fixed to a * * I.. ** ** * * * : *.* * * * * S * *. * * * S * S * * ** S S.. * platform 115 which is rigidly attached to the wall 106 of the cap 100. The pattern 114 comprises a plurality of elements making up an area of the position coding pattern 6, which when imaged, are used by the processor 18 to determine the location of the camera 12 relative to the cap 100.

Due to the snap fitting mechanism 104 and the orientation ridge 108, which locate the lid 100 relative to the pen 8, the position of the nib 10 relative to the cap 100 is known accurately. Therefore from the image captured by the camera 12 when the cap is snap fitted into place the processor 18 is able to obtain information on the position of the camera 12 relative to the cap 100 and hence relative to the pen body and hence relative to the nib.

The memory 22 contains a stored image of an area of the pattern 114 which should be imaged by the camera 12 when the cap 100 has been snap fitted into place assuming there has been no misalignment in the relative position of the nib 10 and camera 12. Information on deviation from this stored image is provided by analysing the part of the pattern 114 which is actually imaged. This provides an indication of how far from the nominal, aligned condition the camera 12 is and hence its current position relative to the nib.

In use, the processor 18 determines from the pattern 6 the position of the nib 10 whenever it is in contact with the document 2 as previously mentioned. To do this the position of the camera 12 relative to the document 2 is determined from the sequence of time stamped images captured by the camera. Then the offset between the nib 10 and camera 12 is used to determine the position of the nib 10, and hence the position of the pen strokes on the document, when each of the images was recorded. Therefore, the current position of the camera 12 relative to the nib 10 is taken into account by the processor 18 in order to include a S *S. S. ** * . S * S *** * * * S : : : * : : : * *.* : ** s correction for the offset between the nib 10 and the camera 12. The pen stroke data can then be sent to the pen stroke interpretation system as previously described. The processor 18 thus uses the most up to date calibration information to determine the required correction for the offset.

In an alternative embodiment, the pen stroke data can be sent via a pen docking station to a suitable pen stroke interpretation system. In this case, the pen is docked at the station after use and information is downloaded onto the interpretation system directly through the docking station. In this case the docking station may include calibration means as previously described. There is then no need for a pen cap and the calibration process can be conveniently carried out whenever the pen is docked.

In a further alternative embodiment, the calibration process can be performed after the sequence of pattern images has been collected and before the pen stroke data is sent.

Also, in an alternative embodiment, there may be many stored images in the memory 22, each stored image equating to a different relative position of the nib 10 to the camera 12. In this case the captured image can be compared by the processor 18 to the stored images in the memory 22 to find a match and thus determine the position of the nib 10 relative to the camera 12 in this way.

Referring to Figure 3, a pen cap 200 including a calibration mechanism according to a further embodiment of the invention is shown. The cap is similar to the cap of the first embodiment in that it has a generally circular cross section along its length, a circumferential ridge 204 on an interior wall 206 and an elongate ridge 208.

* S **. S * * * . S * * **. S S S S * S S IS * S S. S 5 5 S I I * * * : This cap 200 differs from that 100 of the first embodiment in that it has a patch of data encoding pattern 214 (similar to the patch of data encoding pattern 114) mounted on a platform 215 that is moveably attached to the interior wall 206 of the cap 200. The platform 215 is mounted on a flexible elastomeric membrane 220 illustrated schematically in Figure 3, or similar mechanical mechanism that permits movements in any direction in the horizontal plane. The platform 215 has a nib receiving aperture 222, with a shape to receive the nib over expected range of positions, formed therein arranged to receive the nib 10 when the cap 200 is snap fitted into place on the pen 8. In this embodiment, the aperture 222 is tapered to guide the nib 10 into position - the widest part of the aperture 222 is at a side of the aperture facing towards the open end of the cap 200. The purpose of the aperture is to always position the calibration target in the same position referenced to the pen nib, hence niaking the pen nib the reference point of all further calculations of position of the target. The position of the pattern 214 relative to the aperture 222 is fixed therefore the position of the pattern 214 relative to the nib 10 is fixed when the nib 10 is correctly located within the aperture 222.

The processor 18 can then determine the position of the nib 10 relative to the camera 12 in the same way as in the previously described embodiment.

This calibration system of this embodiment can also correct for any displacement of the pen nib relative to the pen body or cap, as they are not used as a reference in the system.

In both embodiments, the pattern 114, 214 is recognised by the processor 18 as calibration pattern and is thereby distinguished immediately from * * S.. * S * * . . p S * * * * I * * * * *Ip *I S ** . S S * * *.: . the pattern 6 on the document 2 to indicate that the calibration process can be commenced.

When the calibration process has ended, embedded software in the pen performs shutdown' of the pen to preserve battery power. Restart of the pen software can be triggered by the image sensor detecting ambient light or any other suitable method. Alternatively, magnets on the pen cap can be used as the trigger.

Alternatively or additionally, a sensor may be provided to sense that the cap 100, 200 has been snap fitted into place and therefore that the calibration process can be commenced. The sensor may be a mechanical sensor associated with the snap fitting mechanism 104, 204 or with the nib receiving aperture 220.

Referring to Figure 6, a position identifying pattern 608, which can be used as the pattern 114 on the target, is made up of a number of dots 630 arranged on an imaginary grid 632. The grid 632 can be considered as being made up of horizontal and vertical lines 634, 636 defining a number of intersections 640 where they cross. The intersections 640 are of the order of 0.3mm apart, and the dots are of the order of 100pm across.

One dot 630 is provided at each intersection 640, but offset slightly in one of four possible directions up, down, left or right, from the actual intersection 640. The dot offsets are arranged to vary in a systematic way so that any group of a sufficient number of dots 630, for example any group of 36 dots arranged in a six by six square will be unique within a very large area of the pattern. This large area is defined as a total imaginary pattern space, and only a small part of the pattern space is taken up by the pattern on the calibration target. By allocating a known area of the pattern space to the calibration target, for example by means * * *SS * * . S * : : : .: : : * S * * S S S * S * .* S of a co-ordinate reference, the document and any position on the patterned parts of it can be identified from the pattern printed on it. An example of this type of pattern is described in WO 01/26033. It will be appreciated that other position identifying patterns can equally be used.

Some examples of other suitable patterns are described in WO 00/73983 and WO 01/71643.

Referring to Figure 4, for each of the embodiments of Figure 2 and 3, if multiple reference images are stored in memory as described above, the calibration process is as follows. When the calibration process is commenced (e.g. triggered by the pressure sensor 20 detecting the pen nib 10 entering into the aperture 222, and then the detection of the special calibration target or special area of pattern 214), the processor 18 instructs the camera 12 to capture a test image at step 402. The test image is then sent to the processor 18 which interrogates the memory 22 which contains a database of stored reference images at step 404. At step 406, the processor 18 uses a comparison algorithm to match the test image with one of the stored reference images. Each of the stored reference images equates to a known relative position between the nib 10 and the camera 12 - the processor 18 at step 408 thus determines the current relative position.

If only a single reference image is stored in memory, the test image is simply compared with the reference image and the linear and rotational offset of the test image from the reference image determined. This offset can then be used to determine the current position of the nib 10 relative to the camera 12.

Referring to Figure 5, when only a reference position and orientation of the camera and nib are stored in memory, the calibration process is as

S S

* S **. * S S * : : : *.. ** * ** S S * * S S S * I S ** *S S S.. * follows. When the calibration process is commenced, the processor 18 instructs the camera 12 to capture a test image at step 502. The test image is then sent to the processor 18 which analyses the test image at step 504 to determine the position and orientation of the camera relative to the pattern. These are compared to the reference position and orientation to determine the offset in position of the camera from a nominal position. The relative position of the nib 10 and the camera 12 are therefore determined directly from the unique data encoding pattern on the test image, using the fact that the position of the nib 10 relative to the pattern is known.

For all embodiments of the invention in which a stored image is compared with the test image during the calibration process, the stored image is initially obtained for a known relative position between the nib 10 and camera 12. This is most conveniently achieved when the pen 8 is manufactured i.e. "in factory". At this time, the relative position is accurately predetermined or measured and a captured image of pattern inside the pen cap is used as the stored reference image (equating to the known relative position).

For embodiments in which the processor 18 determines the relative position by analysing the unique data encoding pattern without comparison to a stored image, an image may still be captured in a similar manner to that mentioned in the previous paragraph so that the processor 18 can determine from it the reference position and orientation.

Alternatively the reference position and orientation can be entered directly into the pen's memory, assuming that the initial position of the pattern in the lid can be accurately controlled. Any subsequently captured image is analysed to determine, from the subsequently imaged S * S * . . . * * * * .:. * * * * * S S * * S. * * S. S S S * * S S * * * *5 S S.. * pattern, the lateral and/or rotational offset from the initial position of nib relative to camera 12.

Referring to Figure 7 in a further embodiment of the invention the cap is circular in section and can therefore be placed in any orientation about its axis on the pen. The pen has a corresponding portion of circular section onto which the lid fits, and the camera is offset from the axis of this portion. The position identifying pattern 608 is provided in an annular area 650 of the calibration target. In this embodiment, when the cap is in place over the pen, the camera 12 will be in a position to image a part of the disc shaped area 650. Which part is imaged will depend on its rotational position relative to the pen cap. The thickness of the disc shaped area is sufficient to accommodate for misalignments between the relative position of the nib and camera.

During calibration, the camera 12 captures an image 660 when the cap is in place over the nib. The processor is able to analyse the image and determine from the unique pattern thereon the position and orientation of the camera relative to the nib since the position of the nib relative to the pattern 650 is known as with previous embodiments and analysis of the image allows determination of the absolute position of the camera 12. It is also possible for the processor 18 to determine the distance of the camera 12 from the pattern 650 by measuring the scale of the pattern in the image. If the camera is further from the target the scale of the pattern will appear smaller in the image 660 than if it is close to the target.

The lateral, vertical and rotational position of the camera relative to the target can thus be determined and used to correct for variations in the camera 12 to nib 10 offset.

S S

* S **. S S S * S S S S *SS S S S S * I S S * *. * S S. S S S * * 0 * S S ** S. S S.. S S In another alternative embodiment in which the pen cap does not require an alignment ridge, referring to Figure 8, the calibration target has a circle 850 formed from a solid line marked on it. When the pen cap is in place over the nib, the camera 12 is arranged to capture an image 860 containing a part of the circle 850. Since the radial position of the nib relative to the circle 850 is known when the cap is in place over the nib, the camera 12 is positioned such that the captured image 860 should show, for a nominal case where there is no misalignment between the nib and the camera, an image where the part of the circle which has been imaged passes through predetermined co- ordinates of the captured image e.g. through the centre of the captured image with a particular inclination and radius of curvature. The processor 18 analyses the captured image to determine deviations in lateral position, inclination and radius of curvature to obtain information on the relative lateral position, rotational portion and vertical position of the nib relative to the camera. The radius of curvature of the part of the circle in the captured image 860 provides information on the distance of the camera to the circle 850 and since the relative position between the nib and circle 850 is known, the deviation in the vertical height of the camera can be determined. The further away the camera is from the target, the smaller the radius of curvature will appear to be.

In the embodiments of Figures 7 and 8, the pen cap does not have an alignment ridge 108, 208 as in the previously described embodiments.

Instead the pattern located on the target in the pen cap contains enough data for the processor 18 to determine the relative position of the nib 10 and camera 12 irrespective of the orientation of the pen nib 10 within the cap. In this way construction of the cap is made easier - it is more economical, more design freedom is provided and the error due to rotational misalignment is negated.

S S

* S **. * S S S * * S * * **. * * a a * S S.5 S S * . a * a S S S * S S. *

SSS S

Referring to Figure 9, for each of the embodiments shown in Figures 7 and 8, the calibration process is as follows. When the calibration process is commenced, the processor 18 instructs the camera 12 to capture a test image at step 902. Next, the processor 18 calculates the position and orientation of the camera from the captured image at step 904. For the embodiment shown in Figure 7, this is achieved by analysing the unique portion of dot pattern which is imaged. As described above, from any area of the pattern of Figure 6 of at least a predetermined size, the position and orientation of the camera relative to the target can be determined from the features of the pattern. For the embodiment shown in Figure 8, this is achieved by analysing the imaged part of the circle and comparing its position and orientation within the imaged area to an expected position and orientation, which is the same regardless of the orientation of the lid relative to the pen. At step 906, the processor 18 uses a comparison algorithm to compare the camera position with the expected position and orientation for a known positional and orientational offset relative to the nib. Where the term relative position has been used in relation to step 906, it will be understood that this includes lateral, vertical and orientational position. Next at step 908, the processor 18 uses the information obtained from the comparison algorithm at step 906 to determine the relative position (lateral and vertical) and orientation between the nib and the camera.

In other embodiments of the invention, the calibration target may not be a location encoding pattern or a dot pattern. It may be any specially designed suitable calibration target to calibrate vertical, lateral and rotational offsets in the system. For example, it may be a representation of a square whose offset can be determined by comparing a reference image of the square with a captured image of the square. a

* * ass * * S 5 * * 5 * S *. * * S a * * * a. I Is S S S * * * S S S 5 5* 5 555 5 In other embodiments of the invention, the calibration system can be used to detect an offset which is unacceptable i.e. so large that the pen will not be able to function properly. This could occur where the vertical distance between the camera and the data encoding pattern which is being read from a document has become so large due to the misalignment that the data encoding pattern on the document cannot be read sufficiently well.

Similarly, if the lateral offset or the rotational offset becomes too large, the camera may not be able to capture an image which includes enough of the calibration target, for example, when the pen cap is in place over the nib. In this case, an indication that the position of markings on the document made by the pen cannot be made and that the pen cannot be calibrated will be provided.

A user can be warned of this by a visible or audible warning mechanism provided on the pen such as an LED or buzzer or a vibrator. The warning mechanism may automatically prevent use of the pen, at least until the pen is repaired. This warning mechanism can also be used to determine that certain parts of the camera/other parts of the pen are coming out of alignment often or more often than other parts and could provide advance warning of a mechanical failure.

Also in other embodiments, the resilient mounting of the calibration target within the pen cap may be achieved by using mechanical mounting means such as a spring or springs instead of a rubber material. S *

6 *;S * * S P : : * : *. : : 5* S P P 5 P * 54 6 4* P 5

Claims

1. A digital pen comprising a nib arranged to be moved over a document, an imaging system arranged to image an area of data encoding pattern on the document and calibration means arranged to determine the position and orientation of the imaging system relative to the nib.

2. A digital pen according to Claim 1, wherein the calibration means comprises a calibration target and locating means arranged to locate the calibration pattern relative to the nib, the imaging system being arranged to image a test area comprising at least part of the calibration target, and a processor arranged to determine the position and orientation of the imaging system relative to the nib by analysing the image of the test area.

3. A digital pen according to Claim 2, wherein the processor is arranged to determine the relative position by determining the position and orientation of the imaging means from the image and comparing it with an expected position and orientation corresponding to a known relative position and orientation of the imaging means and the nib.

4. A digital pen according to Claim 3, wherein the processor is arranged to determine the position and orientation of the imaging means by analysing internal parameters of the imaged area of the target.

5, A digital pen according to Claim 3, wherein the processor is arranged to determine the position and orientation of the imaging means by analysing the position and orientation of part of the target within the imaged area.

6. A digital pen according to Claim 2, wherein the processor is arranged to determine the relative position and orientation by comparing a I I : * : : *

SI I I I

* I * * I. I II.

the test area image with a reference image stored in a memory accessible by the processor.

7. A digital pen according to Claim 6, wherein the reference image comprises an image of at least part of the pattern corresponding to a known relative position between the imaging system and the nib.

8. A digital pen according to any preceding claim further comprising a pen cap arranged to be placed over the nib when the pen is not in use, the calibration target being located within the cap.

9. A digital pen according to Claim 8 further comprising locating means arranged to locate the nib in a predetermined position relative to the pattern when the pen cap is in place over the nib.

10. A digital pen according to Claim 8 or Claim 9, wherein the target is fixedly located inside the cap.

11. A digital pen according to Claim 8 or Claim 9, wherein the target is moveably located inside the cap.

12. A digital pen according to any of Claims 9 to 11, wherein the target is supported on a platform attached to the cap, the locating means is formed in the platform and arranged to cooperate with the nib to locate the target relative to the nib.

13. A digital pen according to Claim 12 when dependent on Claim 11, wherein the platform is resiliently mounted inside the cap.

* * *s* a a : * : : : .1: * S S I I a I * S. I ISo S S

14. A digital pen according to any preceding claim, wherein the calibration means is arranged to determine the position of the imaging system relative to the nib automatically upon each cap or dock of the pen.

15. A digital pen according to Claim 14, wherein the calibration means is arranged to determine the relative position after each use of the pen.

16. A digital pen according to Claim 15, wherein the calibration means is arranged to determine the relative position automatically whenever the pen cap is placed over the nib.

17. A digital pen according to any preceding claim arranged to detect movement of the imaging means to a position in which the pen cannot function correctly, and in response to issue a warning to a user.

18. A method of calibrating a digital pen according to any of Claims 1 to 17 comprising the steps of: locating the nib in a known position and orientation relative to a calibration target; imaging a test area comprising at least part of the calibration target; and determining the position and orientation of the nib relative to the imaging system from the test area image.

19. A digital pen substantially as described herein with reference to any one or more of the accompanying drawings.

20. A method of calibrating a digital pen substantially as described herein with reference to one or more of the accompanying drawings.

* S S 5 S * S * I * * : *** . : : : .* ** *. . S **I *.*