GB2440921A

GB2440921A - A digital pen incorporating an orientation and position sensor and a display projector

Info

Publication number: GB2440921A
Application number: GB0615146A
Authority: GB
Inventors: Guy De Warrenne Bruce Adams; Andrew Arthur Hunter; Robert Thomas Owen Rees
Original assignee: Hewlett Packard Development Co LP
Current assignee: Hewlett Packard Development Co LP
Priority date: 2006-07-31
Filing date: 2006-07-31
Publication date: 2008-02-20
Also published as: GB0615146D0

Abstract

The digital pen 1 incorporates a sensor which may be a camera 4 with infra-red light source, lens system 7 and sensor 6. This generates disposition data in accordance with the pen's disposition relative to a medium surface. Disposition data may comprise position and/or orientation. A display projector 11 projects a display on to the surface. Display controller 14 modifies the display 13 in accordance with the disposition data. Pen orientation data may be used to correct perspective distortion. The pen may be used in conjunction with digital writing paper having a pattern of dots from which pen position may be derived. Pressure sensor 9 detects contact of writing stylus 3 with the writing surface. Pen position data may be used to vary the content of the display with pen position, when the content may be form filling instructions. Alternatively, the content of the display may be related to pen strokes and thus provide feedback to the writer.

Description

1 2440921

DIGITAL PEN

FIELD OF TIlE INVENTION

The invention relates generally to a digital pen, which term is intended to cover a hand-held writing or pointing device intended to be used in connection with a writing or display medium.

BACKGROUND

Many digital pen and paper systems have been proposed. One that is in use is the Anoto system, which can be implemented using a device such as the Logitech 10 2 pen, available from Logitech of 6505 Kaiser Drive, Fremont, Calif. 94555, USA. Generally, using such technology, the pen senses a position determining pattern that has been printed onto a page and an evaluation of the pen's position, and movements, is made using data collected by the pen.

WO 03/046708 discloses a system of this kind. In the known Anoto type arrangements, the pen is connected by a Universal Serial Bus (USB) cable or wirelessly to a processing device such as a mobile telephone or a personal computer. The processing device receives data from the pen and can identify the document which has been marked by the pen. This can result in the processing device determining information about how the document should be handled.

This information may identify an application, perhaps stored on the processing device or held remotely, which enables the information from the pen to be processed.

SUMMARY OF THE PRESENT INVENTION

According to the first aspect of the invention, there is provided a digital pen incorporating a sensor to generate disposition data in accordance with the pen's disposition relative to a medium surface, a display projector to project a display onto said surface and a display controller to modify the display in accordance with the disposition data.

In one embodiment of the invention, the disposition data comprises pen orientation data representing the orientation of the pen, and this data is used to modify the display so as to counter perspective distortion of the display on the medium surface as caused by variation in the angle of incidence of the projection at the surface.

In another embodiment of the invention, the disposition data comprises pen orientation and/or pen position data, where pen position is the position of the tip of the pen, and this data is used to modify the display so as to counter movement of the display as it appears at the medium surface. Thus, the position of content within the display may be varied as the display itself moves. In order to archive this effect, the content is displayed over just a limited portion of the full width of the display.

In yet another embodiment of the invention, the disposition data comprises pen position data, and this data is used to vary the content of the display with position. The content may be related directly to position, for example, the content may relate to form filling instructions where the medium surface is that of a form. however, the content may alternatively be related to pen strokes, as identified through change of pen position, and the content may then constitute feedback to the writer as the pen is used, for example, to project predictive text or spell checking.

The pen is preferably adapted to work with a digital medium having on its surface a position-determining pattern which the pen senses, for example, through a camera, to generate said disposition data. Preferably, the position-determining pattern has a known two-dimensional regularity such as to allow the angle of the pen relative to the surface to be assessed. For example, a camera viewing the pattern would see a perspective view as the pen is tilted from the orthogonal position, and the perspective will be related to the angle of viewing. The angle of the pen can therefore be deduced and used as an output to modify the projected display, for example, to correct perspective distortion. In practice a geometric transform can be used to convert the observed perspective to a projection correction for perspective distortion.

The position-determining pattern is preferably adapted to allow detection of the angular orientation of the pen about its longitudinal axis so that this can be used to modify the projected display and correctly position displayed content.

The position-determining pattern is preferably adapted to allow the absolute position of the pen on the surface to be determined. For example, the pattern may incorporate a coding scheme that is detected by a sensor in the pen and decoded.

In other embodiments of the invention, the pen may be provided with sensors to sense the orientation of the pen in an absolute sense, for example, static accelerometers or tilt angle sensors. Alternative types of sensors may also be provided to detect the absolute position of the pen.

The projected display may be provided by an optical projector, such as a scanning laser projector, or an oscillating optical fibre projector.

The pen may be provided with a writing stylus so that it can write on the medium, but in alternative embodiments of the invention, the pen may simply act as a pointer but otherwise have the same functionality except the ability to make marks directly on the medium.

According to a second aspect of the invention, there is provided a method of communicating with a user of a digital pen, the pen being adapted to generate disposition data in accordance with the pen's disposition relative to a medium surface, and further adapted to project a display onto said surface and to modify the display in accordance with the disposition data.

According to a third aspect of the invention, there is provided a computer program product for communication with a user of a digital pen, the computer program product comprising a computer useable medium having computer executable program code embodied thereon which is operable in association with a computer to respond to disposition data generated by a digital pen in accordance with its disposition relative to a medium surface, and adapted to control a display on the surface as projected by a display projector mounted on said digital pen, the display being modified in accordance with the disposition data.

According to a fourth aspect of the invention, there is provided a computer program comprising computer executable code operable in association with a computer to respond to disposition data generated by a digital pen in accordance with its disposition relative to a medium surface, and adapted to control a display on the surface as projected by a display projector mounted on said digital pen, the display being modified in accordance with the disposition data.

DESCRIPTION OF THE DRAWINGS

The invention will now be further described, by way of example only, with reference to the following drawings, in which: Figure 1 shows a schematic drawing of a pen according to the invention; Figure 2 shows a schematic drawing of the pen of Figure 1 being used in an upright mode; Figure 3 shows a schematic drawing of the pen of Figure 1 being used in a tilted mode; Figure 4 shows variation with tilt angle of the position of content within a display frame; Figure 5 shows an application of the pen of Figure 1 to generate a prompt for form fillings; Figure 6 shows an application of the pen of Figure 1 for predictive text or spelling; and Bgure 7 shows a flow diagram illustrating operation of the pen of Figure 1.

DETAILED DESCRIPTION

A digital pen according to the embodiment of the invention illustrated in Figure 1 comprises an elongate body I with a writing tip 2 at one end, on which is mounted a writing stylus 3 and a camera 4 set up to image an area of a writing surface adjacent to the tip of the pen. The camera 4 comprises a light source 5, such as an infra-red light emitting diode LED, an infra-red sensor 6, and a lens system 7 to collect infra- red light reflected from the writing surface and conduct it to the sensor 6. A processor 8 within the pen body 1 processes image data from the sensor 6. A pressure sensor 9 detects contact of the stylus with the writing surface and triggers operation of the pen.

The pen is adapted for use with digital writing paper having a position-determining pattern on it made up of a number of dots on an imaginary grid of intercepting horizontal and vertical lines. A dot is located at each intersection, but is offset slightly from the intersection in one of four possible directions along the horizontal and vertical lines. The offsetting of the dots is varied in a systematic manner so that any group of dots of a particular size, for example, a group in a square pattern 6x6, is unique, and can be correlated with a coordinate position on the paper. It will be appreciated that the dots arc finely applied and do not have a significant presence to the unaided eye.

When the pen is in use, the processor 8 processes the image of dots which it receives from the sensor 6, and deternines the corresponding position of the tip of the pen on the writing surface. This position information is stored in a memory 10 within the pen body and is used together with successive detenninations of tip position to determine the shape and position of any strokes or marks made by the tip of the pen. An example of a digital pen and digital paper of the type described above is shown in W003/046708. Examples, of position-determining patterns are also described in WOO 1/26033, W000/73983 and WOO 1/71643.

The pen is also provided with a display projector 11 located on the outside of the body I so as to project a light beam 12 towards the tip of the pen so as to form a display 13 on the writing surface adjacent to the tip. The projection display may comprise any known device of an appropriate compact size, such as an oscillating optic fibre display as described in US5727098, or a scanning laser beam display as proposed by Microvision, Inc., or an LCD combined with LED projector.

The display projector 11 is preferably located towards the tip of the pen on the opposite side from the camera 4. A controller 14 is provided within the body of the pen to drive the display projector and control the display 13.

If the pen is used in an upright mode, with the light beam orthogonal to the writing surface 15, as shown in Figure 2, the display 13 on the writing surface appears in an undistorted orthogonal mode, which in this example, is shown as a quadrilateral. However, if the pen is tilted from the upright mode, the display 13 is distorted by a perspective effect, as shown in Figure 3. The lilt plane is the plane of the drawing, and the tilt angle 0 is measured between the pen's longitudinal axis and the writing surface 15. The parallel sides a-c and b-d of the quadrilateral extend perpendicular to the tilt plane, and remain parallel as the pen is tilted.

However, the length of the nearer side a'-c' is shorter than that of the further side b'-d', and therefore the other pair of opposite side a'-b' and c'-d' of the quadrilateral, which began parallel in the orthogonal mode, diverge as the pen is tilted. The display as a whole shifts away from the point of contact P of the stylus 3 with the writing surface 15, and is distorted into a trapezoidal shape and enlarges with reducing tilt angle 0.

The distortions of the display 13 on the writing surface as the pen is tilted cause difficulties in reading the display, and therefore it is preferred to apply a correction to the display so as to counter these distortions.

It will be appreciated that, whilst the process of distortion and correction of the display has been described above in relation to tilt of the pen in a single plane, the same principle can be applied in two mutually orthogonal planes that flit independently in any direction about the point of contact P of the stylus 3.

These changes can be represented mathematically by a perspective transform which maps the coordinates of the planar 2D display in the orthogonal mode to those of the display in the tilted mode. These have been described repeatedly over the years and are commonplace in machine vision. A common reference source is Semple J. and Kneebone G. Algebraic Project ive Geometry. Oxford University Press, 1979. Another description of a suitable perspective transform in principal is described in "A plane Measuring Device", A. Criminisi, I. Reid, A Zisserman (Dept. of Engineering Science, University of Oxford, Proceedings of the British Machine Vision Assoc. Conference 1997), BMVC 97, Session 10, ISBN 0952 18987 9 and which is further explained by the additional document at http://alunmi.media.mit.edu/wrepJjnterpolator/ Specifically, this last reference discloses a perspective transform H suitable for mapping a quadrilateral in a first plane X, Y into a quadrilateral in a second plane x, y, where the first plane X, Y corresponds to the plane of the display at the display projector 11 having a predetermined quadrilateral-shaped frame, and the second frame x, y corresponds to the plane of the writing surface 15 on which the distorted trapezoidal-shaped frame is projected as shown in Figure 3. The mapping is written as: X=Hx which can be rewritten as: XW a h c x YW = d e / y W ghl 1 Thcreforc, such a perspective transform can be used in the controller 14 so as to generate a correction that can be applied to all the display points to counteract the perspective distortion and have the display appear at the writing surface in the orthogonal mode (or any other fixed mode) over a range of tilt angles of the pen.

Preferably, the display is further corrected so that it appears as a fixed printed display or a display that moves in a controlled manner with progressive movement of the stylus of the pen across the writing surface. In order to achieve this, the display is configured so that the content occupies only a portion of the maximum display frame and its position can be varied within the display frame in accordance with movement of the pen. For example, tilting of the pen will cause the display to move and distort on the writing surface, but a position correction related to the tilting movement and the perspective distortion correction is applied by the display controller unit 14 to counter the movement and distortion of the content within the display frame. A nominal angle of tilt may be selected and variations from this angle monitored to drive the position correction of the content.

This control technique is illustrated in Figure 4, which shows in Figure 4a a maximum rectangular display frame corrected for perspective distortion and having the content PROMPT within it. Movement of the pen causes the display frame to move to the right as shown in Figure 4b, and without any further correction to the display, the content PROMPT would move with the display frame. However, as shown in Figure 4b, a correction is applied that moves the content PROMPT left within the frame so that the content PROMPT appears not to have moved. however, this correction cannot be applied any further as the content approaches the margin of the display frame without losing some of the content e.g. continued right movement of the display frame in Figure 4b would cause the first letter "P" to be lost from PROMPT as shown in Figure 4c. In order to counter this problem, the content can be scaled or otherwise rearranged within the frame so that all of it still appears. however, a preferred solution is to reposition the content PROMPT to the opposite, i.e. right-hand margin in Figure 4c, on the assumption that the display will continue to move to the right and the content PROMPT can be maintained for a period of time in this new fixed position as the display continues to move right. This correction technique is appropriate to counter the effect of tilting movement, and especially translational movement of the pen across the writing surface in the normal manner when writing left to right and this motion can be inferred from the task or region of the paper. Translational movement of the pen is therefore detected, as well as tilting movement, and the display controller 14 controls the display in accordance with the sensed translational movement of the pen to move the content in the display in steps, as shown in Figures 4b and 4c. For other uses of the display, it may be more appropriate to adopt a different approach that optimises the characteristics of the display to the particular use. (e.g. freezing motion, continuous motion etc.) Rotary movements of the pen about its own longitudinal axis will also cause the display 13 to rotate about the point P of the stylus 3 on the writing surface. These rotary movements of the display arc preferably countered by sensing these rotary movements of the pen, and using them in the display controller 14 to counter corresponding rotary movements of content within the display. The margins of the display determine how much rotation can be accommodated before the display is disrupted. Therefore the ergonomics of the pen is designed to assist in ensuring the display is correctly orientated for both normal pen grip left and right handed people (i.e. with the display above the pen tip). For those who use a pen in a grip that results in writing from above the line, the display controller uses the rotary information from the alignment of the pen to the paper, and rotates the text in order to display it below the line to avoid obscuration by the pen.

Therefore, in the preferred embodiment of the invention, tilting movement of the pen in all directions, rotational movement of the pen about its longitudinal axis, and translational movements of the pen are all sensed, and appropriate corrections generated by the display controller 14 to modify the display 13 so as to counter perspective distortions and unwanted movements of the content of the display. By countering tilting movements in two mutually orthogonal planes, content in the display can be maintained in a desired position, for example along a line on the writing surface, and be made to move in a controlled manner along the line or as appropriate with translational movement of the pen.

Figure 5 illustrates an example of an application of the invention in which a pen according to the invention is being used to fill in a form composed on digital paper. The pen is shown filling in an entry in a box on the form and is projecting a display 13 containing a prompt to inform or instruct the user. The text PROMPT appears as if it were printed on the paper surface and remains stationary whilst the pen is moved to fill in the box. Alternatively, if the box is too long and exceeds the maximum width of the display, the PROMPT is held stationary at successive positions as the display is moved stepwise across the top of the box.

In an alternative application of the invention, as shown in Figure 6, the display is actually prompting with predictive text or spelling in response to an entry being made by the user.

The display shows ROMPT in response to the user entry "P". In other applications, the display may contain commentary relevant to the user's task as identified through position information of the stylus 3, as detected by the camera 4. The processor 9 decodes the image of the dots on the digital writing paper as viewed by the camera 4, and thereby establishes the stylus position information. This is then crossreferenced with commentary in a database held in a memory within the pen, such as memory 10 or a separate dedicated memory.

Alternatively, the database is held in an external memory accessed via an external connection 15, which may be a wired connection or a wireless connection.

The means used to detect the angle of tilt 0 and angular orientation of the pen may comprise sensors such as tilt angle sensors or static accelerometers. However, both tilt angle and angular orientation are preferably sensed through the image of the pattern of dots on the digital paper as viewed by the camera 4. The pattern has a predetermined format in two dimensions based on a grid layout, as explained above, and therefore an image of the pattern will be subject to perspective distortion and disorientation of the grid, which can be analysed to determine tilt angles and angular orientation of the pen. The necessary image processing may be performed in the processor 8 or a separate internal processor. The derived tilt angle and angular orientation outputs from the processor are then fed to the display controller 14.

It will be appreciated that the determination of the tilt angle from the perspective distortion observed by the camera 4 could involve use of the same or a similar perspective transform as that used to apply the corresponding correction to the display by the display controller 14.

However, it is preferred if possible to use the observed perspective distortion plus an offset due to the fact the camera and nominal axis of the display are not coaxial, as an input to a single transform that generates the corrected output displays, and therefore the division of the processing between the camera processor 8 and the display controller 14 may differ from that described above, and the transform processing may take place oniy in the display controller 14. The typical imaging rate for the camera 4 would be greater than 75fps, which allows enough time for real time correction if there is sufficient processing capability available, Also, if an oscillatory optic fibre type display is used, the correction may be applied in the analogue domain by varying the x, y drive inputs of the fibre.

The pen is further preferably adapted so that it can still operate, but without the writing function, when the stylus 3 is out of contact with the writing surface 13. The pen then needs to know how far the stylus is from the writing surface so that an appropriate adjustment can be made to the correction transform in the display controller 14. The separation distance from the writing surface can be determined by reference to movement of the pen in lifting off the surface having first made contact with it. The pressure sensor 9 detects contact of the stylus with the writing surface, and a sensor, such as a static accelerometer, may be provided within the body of the pen to detect movement away from the writing surface. Also the change in scale of the pattern on the surface can be used to indicate distance. These inputs can be fed to the processor 8 or display controller 14 to trigger the required adjustment to the correction transform.

It will be appreciated that the ability of the pen to operate without necessarily writing on the writing surface, allows it to be used as a pointer to generate an appropriate display to inform or instruct the user. Thus in another embodiment of the invention, the writing function may be omitted altogether, and the pen effectively becomes a pointer.

The overall mode of operation of the pen is illustrated by the flow diagram in Figure 7. A user starts at Step 20 by picking up the pen and powering it on using a manual switch built into the pen or removing the cap of the pen to operate a switch indirectly. The user then brings the stylus 3 into contact with the surface of the digital paper at Step 21, which operates the pressure sensor 9 to trigger image processing in the processor 8 and the display controller 14. Whilst the stylus 3 is maintained in contact with the digital paper, the processor 8 operates at Step 22 to decode the image from the camera 4 to determine the position of the stylus on the paper, the tilt angles and angular orientation of the pen about its longitudinal axis, and the stroke made by the pen. The position information and stroke information from Step 22 are then used in Step 23 to determine the content of the display which is to be projected by the display projector 11 onto the paper. The content is then supplied to the display controller 14, which at Step 24 applies a correction to the display which includes the selected content. The corrected display is then applied to the display projector 11 in Step 25.

If at any time, the stylus is lifted away from the paper for a period of more than 2Oms for instance, then processing of data by the processor 8 and display controller 14 ceases until contact is made again as sensed by the pressure sensor 9. However, if the pen is adapted to operate in a "hover" mode, when lifted out of contact with the paper, then Step 21 is modified to include detecting of the pattern on the digital paper, and this is sufficient to maintain the pen in operation until the pattern is no longer detected. Processing is then stopped and Step 21 is repeated.

Claims

Claims 1. A digital pen incorporating a sensor to generate disposition

data in accordance with the pen's disposition relative to a medium surface, a display projector to project a display onto said surface and a display controller to modify the display in accordance with the disposition data.

2. A digital pen as claimed in claim 1 in which the sensor is adapted to generate disposition data comprising pen orientation data representing the orientation of the pen, and in which this data is used by the display controller to modify the display so as to counter perspective distortion of the display on the medium surface as caused by variation in the angle of incidence of the projection at the medium surface.

3. A digital pen as claimed in claim 1 or 2 in which the sensor is adapted to generate disposition data comprising pen orientation [and/or pen position] data, corresponding to the position of the tip of the pen, and this data is used by the display controller to modify the display so as to counter movement of the display as it appears at the medium surface.

4. A digital pen as claimed in claim 3 in which the display projector and controller are such that the display occupies a frame, content of the display occupies only a limited portion of the frame, and the position of content within the display is varied by the controller as the display itself moves.

5. A digital pen as claimed in claim I in which the sensor is adapted to generate disposition data comprising pen position data, and this data is effective to vary the content of the display with position.

6. A digital pen as claimed in claim 5 in which the content of the display is controlled by the controller so that it is related to pen position.

7. A digital pen as claimed in claim 6 in which the content relates to form filling instructions for display on a medium surface comprising a form.

8. A digital pen as claimed in claim 1 in which the sensor is adapted to generate disposition data comprising pen position data and the pen incorporates a processor to analyse the pen position data and to identify pen strokes, which pen strokes determine the content of the display and thereby feedback to the user as the pen is used.

9. A digital pen as claimed in any one of the preceding claims which is adapted to work with a digital medium having on its surface a position-determining pattern which the sensor senses to generate said disposition data.

10. A digital pen as claimed in claim 9 in which the position-determining pattern has a known two-dimensional regularity and the pen incorporates a processor to identify the pattern from the disposition data so as to assess the angle of the pen relative to the surface.

11. A digital pen as claimed in any one of the preceding claims in which the sensor comprises a camera.

12. A digital pen as claimed in claim 2 in which the display controller uses a geometric transform to generate a projection correction for perspective distortion.

13. A digital pen as claimed in claim 9 in which a processor analyse s the position-determining pattern to detect the angular orientation of the pen about its longitudinal axis, and in which the display controller modifies the projected display and correctly positions displayed content in accordance with the angular orientation.

14. A digital pen as claimed in claim 9 in which a processor analyse s the position -determining pattern to allow the absolute position of the pen on the surface to be determined.

15. A digital pen as claimed in any of claims 1 to 8 provided with sensors to sense the orientation of the pen in an absolute sense.

16. A digital pen as claimed in any of the preceding claims in which the display projector comprises a scanning laser projector or an oscillating optical fibre projector.

17. A digital pen as claimed in any of the preceding claims provided with a writing stylus so that it can write on the medium.

18. A method of communicating with a user of a digital pen, the pen being adapted to generate disposition data in accordance with the pen's disposition relative to a medium surface, and further adapted to project a display onto said surface and to modify the display in accordance with the disposition data.

19. A computer program product for communication with a user of a digital pen, the computer program product comprising a computer useable medium having computer executable program code embodied thereon which is operable in association with a computer to respond to disposition data generated by a digital pen in accordance with its disposition relative to a medium surface, and adapted to control a display on the surface as projected by a display projector mounted on said digital pen, the display being modified in accordance with the disposition data.

20. A computer program comprising computer executable code operable in association with a computer to respond to disposition data generated by a digital pen in accordance with its disposition relative to a medium surface, and adapted to control a display on the surface as projected by a display projector mounted on said digital pen, the display being modified in accordance with the disposition data.

21. A digital pen incorporating a sensor to generate disposition data in accordance with the pen's disposition relative to a medium surface, a display projector to project a display onto said surface, and a display controller to modify the display in accordance with the disposition data, wherein the sensor is adapted to generate disposition data comprising pen orientation data representing the orientation of the pen, and in which this data is used by the display controller to modify the display so as to counter perspective distortion of the display on the medium surface as caused by variation in the angle of incidence of the projection at the medium surface.

22. A digital pen adapted to work with a digital medium having on its surface a position-determining pattern, the pen incorporating a sensor to generate disposition data, a display projector to project a display onto said surface, and a display controller to modify the display in accordance with the disposition data so as to counter perspective distortion of the display on the medium surface as caused by variation in the angle of incidence of the projection at the medium surface.