KR100552018B1 - Orientation sensing device - Google Patents

Abstract

When the arrangement or movement in the relationship with the surface is detected by the sensing device, generating direction data indicating the direction of the sensing device in relation to the surface having the encoded data indicating the orientation A sensing device, comprising: a housing; Direction sensing means configured to generate the direction data using at least some of the encoded data; And communication means configured to communicate said direction data to a computer system.

Description

Orientation sensor {ORIENTATION SENSING DEVICE}

The present invention relates to a method, system and apparatus for interacting with a computer, and more particularly to a sensing device for sensing its orientation to a surface when a movement or placement is made in relation to the surface.

The present invention has been developed primarily with the aim of allowing a large number of distributed users to obtain interactive printed matter at any time via a high speed network color printer so that they can interact with network information via prints and optical sensors. Although the present invention will be described hereinafter in most respects in connection with such use, the present invention is not limited to use in this field.

[Pending application]

Numerous methods, systems, and apparatus related to the present invention are disclosed in the following pending applications / granted patents filed by the applicant or the assignee of the present invention concurrently with the present application:

09/721895, 09/721894, 09/722174, 09/721896, 09/722148, 09/722146,

6826547, 6741871, 09/722171, 09/721858, 09/722142, 6788982,

09/722141, 6788293, 09/722147, 09/722172, 6792165, 09/722088,

09/721862, 6530339, 6631897

The disclosures of these pending applications are cross-referenced herein.

A number of methods, systems and apparatuses associated with the present invention are disclosed in the following pending applications / granted patents filed Oct. 20, 2000 by the applicant or the assignee of the present invention concurrently with the present application:

09/693415, 09/693219, 6813558, 09/693515, 6847883, 09/693647,

09/693690, 09/693593, 6474888, 6627870, 6724374, 09/693514,

09/693301, 6454482, 6808330, 6527365, 6474773, 6550997

The disclosures of these pending applications are cross-referenced herein.

A number of methods, systems and apparatuses associated with the present invention are disclosed in the following pending applications / granted patents filed on September 15, 2000 by the applicant or the assignee of the present invention concurrently with the present application:

6679420, 09/663599, 09/663701, 6720985

The disclosures of these pending applications are cross-referenced herein.

Numerous methods, systems, and apparatus related to the present invention are disclosed in the following pending applications / granted patents filed on June 30, 2000 by the applicant or assignee of the present invention concurrently with the present application:

6824044, 09/608970, 6678499, 09/607852, 09/607656, 6766942,

09/609303, 09/610095, 09/609596, 09/607843, 09/607605, 09/608178,

09/609553, 09/609149, 09/608022, 09/609232, 09/607844,

6457883, 6831682, 09/607985, 6398332, 6394573, 6622923

The disclosures of these pending applications are cross-referenced herein.

Numerous methods, systems, and apparatus related to the present invention are disclosed in the following pending applications / granted patents filed May 23, 2000 by the applicant or the assignee of the present invention concurrently with the present application:

09/575197, 09/575195, 09/575159, 09/575132, 09/575123, 6825945,

09/575130, 09/575165, 6813039, 09/575118, 09/575131, 09/575116,

6816274, 09/575139, 09/575186, 6681045, 6728000, 09/575145,

09/575192, 09/575181, 09/575193, 09/575183, 6789194,

09/575150, 6789191, 6644642, 6502614, 6622999, 6669385,

6549935, 09/575187, 6727996, 6591884, 6439706, 6760119,

09/575198, 6290349, 6428155, 6785016, 6870966, 6822639,

6737591, 09/575154, 09/575129, 6830196, 6832717, 09/575189,

09/575162, 09/575172, 09/575170, 09/575171, 09/575161, 6428133,

6526658, 6315699, 6338548, 6540319, 6328431, 6328425,

09/575127, 6383833, 6464322, 6390591, 09/575152, 6328417,

6409323, 6281912, 6604810, 6318920, 6488422, 6795215,

09/575109, 6859289

The disclosures of these pending applications are cross-referenced herein.

In recent years, a user of a computer system typically interacts with the system using a monitor for displaying information and a keyboard and / or mouse for entering information. Such an interface is powerful but relatively bulky and portable. Information printed on paper is easier to read and more portable than the information displayed on a computer monitor. However, unlike keyboards and mice, pens on paper generally lack the ability to interact with computer software.

It is an object of the present invention to combine the advantages of a pen on a paper and a computer system.

According to the present invention, when arrangement or movement is made in relation to a surface, direction data indicating the direction of the sensing device in relation to the surface having encoded data indicating orientation when detected by the sensing device A sensing device for generating a

A housing;

Direction sensing means configured to generate the direction data using at least some of the encoded data; And

And a communication means configured to communicate said direction data to a computer system.

In one of the embodiments, the directional data indicates at least one of yaw, pitch, and roll of the housing in relation to the surface.

Preferably, the sensing device further comprises motion sensing means for generating movement data when the sensing device moves in relation to the surface, wherein the communication means is configured to read the movement data. It is preferably configured to communicate with a computer system.

In addition, when the sensing device is disposed or moved in relation to a region of the surface, the sensing device detects region identification data indicating an identity of the region by using at least some of the encoded data. And further comprising region identity sensing means, said communication means being configured to communicate said region identifier data to said computer system.

Preferably, the direction detecting means dynamically detects the direction of the housing in relation to the surface as the housing moves. The housing may have a long shape that the user can wear. In one of the embodiments, the housing may have the shape of a pen. The housing may be provided with a marking nib for marking on the surface, but is not essential.

By simultaneously capturing direction and movement data, the system can be used to verify a person's signature. Alternatively, the housing may be used as a joystick, driven by a direction signal that is measured dynamically. For example, such joysticks can be used with three-dimensional software applications. It should be noted that it is not essential for the direction detecting means to sense the direction of the housing in all directions in three dimensions. Some applications may not require three-dimensional orientation information, so it may be sufficient to detect only the pitch. For example, the housing can be used to linearly control one side of the device, such as the amount of light in the lamp or the volume of the speaker by varying the pitching between 0 ° and 90 °.

Rolling, pitching and yawing can be calculated by detecting perspective distortion and rotation of the encoded data.

First, it defines the action between the application and removal of pressure as a 'stroke' of hand gestures, which can be used to determine when the device is first applied to the surface and when it leaves the surface. Pressure data information is time stamped.

Preferably the device is a separate implementation comprising suitable means as described above. It may be in any shape, but preferably in the form of a stylus or a pen.

Preferably, the device may incorporate a marking nib for marking handwritten information on the surface, but this is not essential.

Preferably the device is intended to interact with a computer system that is controllable and, in the case of the device, can handle handwritten information (whether drawing or writing) authorized by the user. Preferably, the sensing device is arranged to provide device identification information that uniquely identifies the device. The computer system is thus able to identify the device.

The features and advantages of the present invention will become apparent from the description of the exemplary embodiments described below with reference to the accompanying drawings.

Preferred and other embodiments of the invention are described as exemplary methods without limiting the invention with reference to the accompanying drawings in which: FIG.

1 is a schematic diagram showing a relationship between an example of a printed netpage and its online page specification;

2 is a schematic diagram illustrating interactions between a netpage pen, a netpage printer, a netpage page server and a netpage application server;

3 illustrates a set of netpage servers and printers interconnected via a network;

4 is a schematic diagram showing the high level structure of a printed netpage and its online page specification;

5A is a plan view showing the structure of a netpage tag;

FIG. 5B is a plan view showing a relationship between a set of tags shown in FIG. 5A and a netpage pen-shaped netpage detecting device area;

6A is a plan view showing another structure of a netpage tag;

FIG. 6B is a plan view showing a relationship between a set of tags shown in FIG. 6A and a region of a netpage sensing device having a netpage pen shape; FIG.

FIG. 6C is a plan view showing nine arrangements of tags shown in FIG. 6A, in which a target is shared between neighboring tags

6D is a plan view showing the arrangement and rotation of the four codeword symbols of the tag shown in FIG. 6A;

7 is a flowchart of a tag image processing and decoding algorithm;

8 is a perspective view showing a netpage pen and a tag-sensing field-of-view cone associated therewith;

9 is an exploded perspective view of the netpage pen shown in FIG. 8;

FIG. 10 is a schematic block diagram of a pen controller for the netpage pen shown in FIGS. 8 and 9;

11 is a perspective view of a wall mounted netpage printer,

12 is a vertical sectional view of the netpage printer of FIG. 11;

12A is an enlarged partial view of 12 showing a cross section of duplexed print engines and a glue wheel assembly;

FIG. 13 is a detailed view of the ink cartridge, ink, air and adhesive paths, and the print engine of the netpage printer of FIGS. 11 and 12;

14 is a schematic block diagram of a printer controller for the netpage printer of FIGS. 11 and 12;

FIG. 15 is a schematic block diagram of a dual print engine controller and a Memjet ™ print head associated with the printer controller shown in FIG. 14;

16 is a schematic block diagram of the print engine controller shown in FIGS. 14 and 15;

FIG. 17 is a perspective view of a single Memjet (trademark) printing element used in the netpage printer of FIGS. 10-12, etc .;

18 is a perspective view of a portion of the arrangement of the Memjet (trademark) printing elements,

FIG. 19 is a perspective view showing an operating cycle of the Memjet (trademark) printing element shown in FIG.

20 is a perspective view showing a portion of a page width of a Memjet (trademark) print head,

21 is a schematic diagram of a user class diagram,

22 is a schematic diagram of a printer class diagram;

23 is a schematic diagram of a pen class diagram,

24 is a schematic diagram of an application class diagram;

25 is a schematic diagram of a document and page description class diagram,

26 is a schematic diagram of a document and page ownership class diagram,

27 is a schematic diagram of a terminal element specialization class diagram;

28 is a schematic diagram of a static element specialization class diagram,

29 is a schematic diagram of a hyperlink element class diagram,

30 is a schematic diagram of a hyperlink element specialization class diagram,

31 is a schematic diagram of a hyperlinked group class diagram,

32 is a schematic diagram of a form class diagram,

33 is a schematic diagram of a digital ink class diagram,

34 is a schematic diagram of a field element specialization class diagram;

35 is a schematic diagram of a check box field class diagram,

36 is a schematic diagram of a text field class diagram,

37 is a schematic diagram of a signature field class diagram,

38 is a flowchart of an input processing algorithm;

38A is a detailed flowchart of one step of the flowchart of FIG. 38;

39 is a schematic diagram of a page server command element class diagram;

40 is a schematic diagram of a resource description class diagram,

41 is a schematic diagram of a favorites list class diagram,

42 is a schematic diagram of a history list class diagram,

43 is a schematic diagram of a subscription delivery protocol,

44 is a schematic diagram of a hyperlink request class diagram,

45 is a schematic diagram of a hyperlink activation protocol;

46 is a schematic diagram of a form submission protocol,

47 is a schematic diagram of a commission payment protocol;

48 is a schematic diagram of a set of radial wedges forming a symbol,

49 is a schematic diagram of ring A and B symbol assignment schemes,

50 is a schematic diagram of a first ring C and D symbol assignment scheme,

51 is a schematic diagram of a second ring C and D symbol assignment scheme,

52 is a schematic diagram of a triangular tag packing;

53 is a perspective view of a icosahedron,

54 is a perspective view of an octahedron geodesic at frequency 3,

55 is a schematic diagram of a minimum tag spacing,

56 is a schematic diagram of a minimum tag spacing without duplication;

57 is a schematic view of the first case of tag insertion;

58 is a schematic view of a second case of tag insertion;

59 is a schematic view of the third case of tag insertion;

60 is a schematic view of the fourth case of tag insertion;

61 is a schematic diagram of the pen direction in relation to the surface;

62 is a schematic representation of pen pitching geometry,

63 is a schematic representation of pen rolling geometry,

64 is a schematic diagram of a pen coordinate space showing a physical axis and an optical axis of a pen;

65 is a schematic representation of the bent nib geometry;

66 is a schematic diagram of the interaction between sampling frequency and tag frequency,

67 is a schematic representation of the pen optical path,

68 is a flowchart of a stroke capture algorithm;

69 is a schematic diagram of an untreated digital ink class diagram,

70 is a table containing the first to tenth equations,

71 is a table containing the 11th to 20th equations,

72 is a table containing equations 21-26;

73 is a table containing equations 27-34;

74 is a table containing equations 35-41;

75 is a table containing the 42nd through 44th equations,

76 is a table containing equations 45-47;

77 is a table containing equations 48-51;

78 is a table containing equations 52-54;

79 is a table containing equations 55-57;

80 is a table containing equations 58-59;

81 is a table containing the 60th to 63rd equations,

82 is a table containing the sixty-fourth to seventy-fourth equations,

83 is a table containing the 75th to 86th equations,

84 is a table containing equations 87-99;

85 is a table including equations 100-111;

86 is a table containing equations 112-120;

87 is a table containing equations 121-129,

88 is a table containing a set of weighted forms of the sixty-seventh to seventy-first equations;

89 is a first part of a table including conditions and special handling for zero pitch and zero rolling,

FIG. 90 is a second portion of the table of FIG. 89.

 Note: Memjet is a registered trademark of Silverbrook Research Pty Ltd, Australia.

In a preferred embodiment, the present invention is configured to operate in a netpage network computer system, with a detailed overview as follows. It will be appreciated that not all implementations necessarily include all or most of the specific details and extensions discussed below in connection with the underlying system. However, the system is described in its most complete form to exclude the need for an external literature reference when understanding the context in which preferred embodiments and aspects of the invention operate.

In summary, the preferred form of a netpage system utilizes a computer interface in the form of a mapped surface, ie a physical surface that contains a reference to a map of the surface maintained within the computer system. The map reference is queried by an appropriate sensing device. Depending on the particular implementation, the map reference may be encoded either visually or invisibly, and may be defined in such a way that it presents a clear map reference in both of that map and in different maps for local lookup on the mapped surface. The computer system may include information regarding features on the mapped surface, such information may be retrieved based on a map reference supplied by the sensing device used at the mapped surface. The retrieved information can thus take the form of operations initiated by a computer system on behalf of the operator in response to the operator's interaction with the surface features.

In a preferred embodiment, the netpage system relies on the generation of netpages and human interaction with them. Netpages are pages of text, graphics, and images printed on ordinary paper, but act like interactive webpages. The information is encoded on each page using ink that is essentially invisible to the human eye. However, the ink and thus encoded data can be sensed by an optically imaging pen and transmitted to the netpage system. Other substrates other than paper may also be used. Since the encoded information in the preferred embodiment is an infrared absorbing ink, an infrared reactive optical sensor can be used. If desired, other wavelengths or sensing techniques may be used. Magnetic ink and sensors may be used as an example.

In a preferred embodiment, the active button and hyperlink on each page can be clicked with the pen to request information from the network or to signal a preference to the network server. In one example, text written by hand on a netpage is automatically converted into computer text that can be recognized by the netpage system and entered into a form. In another embodiment, the signature recorded on the netpage is automatically verified to allow secure transaction of electronic commerce transactions.

As shown in FIG. 1, the printed netpage 1 may represent an interactive format that may be physically written on the printed page by the user and may be “electronically” filled by communication between the pen and the netpage system. . The example represents a "application" form that includes a name and address line and a submit button. The netpage consists of graphic data 2 printed using visible ink and encoded data 3 printed as a set of tags 4 using invisible ink. The corresponding page specification 5 stored on the netpage network describes each element of that netpage. In particular, it describes the format and spatial range (area) of each interactive element (ie text box or button in the example), allowing the netpage system to correctly interpret input through the netpage. The submit button 6 has, for example, an area 7 corresponding to the spatial extent of the corresponding graphic 8.

As shown in Fig. 2, the Netpage pen 101 (shown in the preferred embodiment in Figs. 8 and 9, described in more detail below) is a netpage printer 601, for home, office or Works with mobile printing devices. The pen is wireless and securely communicates with a Netpage printer via a short-range radio link 9. The pen may be connected to the system using a wire or infrared transceiver if desired, but these alternatives are limited in ease of use.

The netpage printer 601 (shown in the preferred embodiment in FIGS. 11-13 and described in more detail below) is a personalized newspaper, magazine, catalog, printed in high quality as an interactive netpage periodically or on command. Deliver brochures, brochures and other publications. Unlike personal computers, netpage printers are devices that can be wall-mounted, for example, near the kitchen of the user, where breakfast news is first consumed, near the breakfast table, or near the point where the generation begins the day. Tabletop, desktop, portable and miniature versions are also available.                 

Netpages printed at the point of consumption combine the ease-of-use paper with the timeliness and interactivity of interactive media.

As shown in FIG. 2, the netpage pen 101 interacts with the encoded data on the printed netpage 1 and communicates the interaction with the netpage printer via a short range wireless connection 9. The printer 601 sends the interaction to the relevant netpage page server 10 for interpretation. In the proper environment, the page server sends the corresponding message to the computer application running on the netpage application server 13. The application server can then send a response to be printed on the printer.

The netpage system is made more convenient in the present preferred embodiment by being used with a high speed microelectromechanical system (MEMS) based inkjet (MetJet) printer. In a preferred embodiment of the present technology, relatively high speed and high quality printing can be provided to consumers. In a preferred embodiment, the netpage publication has the physical characteristics of a traditional news magazine, such as a pair of letter-size glossy pages printed in two-sided, colored, for easy transportation and comfortable handling.

Netpage printers take advantage of increasing broadband Internet access possibilities. Cable service is available in 95% of homes in the United States, and 20% of cable modem services that provide broadband Internet access are already available. Netpage printers can operate on slower connections, but they also take longer delivery times and lower image quality. In fact, the Netpage system is also capable of using existing home inkjet and laser printers, but the system runs slower and therefore less satisfactory from the consumer's point of view. In another embodiment, the netpage system is operated on a private intranet. In another embodiment, the Netpage system is run on a single computer or on a device activated by a computer, such as on a printer.

The netpage publishing server 14 on the netpage network is configured to send print-quality publications to a netpage printer. Periodic publications are automatically sent to subscriber netpage printers via pointcasting and multicasting Internet protocols. Personalized publications are filtered and formatted according to each user profile.

The netpage printer is configured to support any number of pens, and the pen can work with any number of netpage printers. In a preferred embodiment, each netpage pen has a unique identifier. One household may have a set of colored Netpage pens each assigned to a family. This results in each user having a distinct profile with respect to the netpage publishing server or application server, assuming only designated pens are used by each family member. However, as described below, other means may be used to identify the user.

The Netpage pen can also be registered with a netpage registration server 11 and linked to one or more payment card accounts, thereby ensuring that e-commerce payments are secured using the Netpage pen. The Netpage registration server verifies the identity of the user to the e-commerce server by comparing the signature obtained by the Netpage pen with the signature already registered, and other biometric information may also be used to verify the identity. One variation of the Netpage pen includes fingerprint scanning, which is validated in the same way by the Netpage registration server.

While netpage printers can deliver periodicals, such as morning newspapers, without user intervention, netpage printers can be configured not to deliver unsolicited junk mail. In a preferred embodiment, the netpage printer receives only periodicals from an authenticated source, whether subscribed or unsubscribed. In this regard, a netpage printer is different from a fax or e-mail account that is exposed to a junk mailer who knows a phone number or e-mail address. Or the entire system may be visible to external users or each user may be provided with the ability to expose their printer to external users. This can also send junk e-mail as a way to select external users.

1. NETPAGE SYSTEM ARCHITECTURE

Each object model of the system is described using a Unified Modeling Language (UML) class diagram. A class diagram consists of a set of object classes connected by a relationship, where the two relationships of interest are association and generalization. Associations represent a kind of relationship between objects, that is, instances of a class. Generalization relates to the actual class and can be understood in the following way-if a class is a collection of all objects of that class, and class A is a generalization of class B, then B is simply It is a subset.

Each class is represented by a rectangle with the name of the class. The class diagram includes an attribute list of the class, separated by a horizontal line from the name, and an operation list of the class, separated by the attribute list and the horizontal line. However, operations are not modeled in the class diagram below.

A union is drawn as a line connecting two classes, optionally with the multiplicity of the combinations at both ends. The default multiplicity is one. An asterisk (*) indicates "many" multiplicity, that is, zero or greater. Each association may optionally be indicated by its name, and the role of the class to which it is optionally associated may also be indicated at both ends. Open diamond refers to an aggregation association ("part of-") and is shown at the aggregation end of an association line.

The generalization relationship ("-is-") is represented by a solid line connecting the two classes, with the triangles (open triangles) shown at the generalization end.

When a class diagram is decomposed into a plurality of figures, the class is represented by a dotted line except for the main diagram that defines it. A class is only represented with properties where it is defined.

1.1 NETPAGES

Netpages are the foundation on which netpage networks are built. Netpages provide a paper-based user interface to published information and interactive services.

One netpage comprises a printed page (or other surface area) that is invisibly tagged with respect to the online specification for that page. The tag may be printed on the surface of the page, present on the bottom layer of the page, or otherwise incorporated into the page. The online page description is maintained continuously by the netpage page server. The page specification describes the visual layout and content of the page, including text, graphics, and images. The page specification also describes the input elements on that page, including buttons, hyperlinks, and input areas. The page specifications of different netpages may share elements such as images, even if the netpages (and their associated page specifications) are visually different. The page specification for each netpage contains references to these common elements. The netpage is marked by a netpage pen on the surface, and at the same time a mark is captured and allowed to be processed by the netpage system.                 

Multiple netpages can share the same page specification. However, in the case of receiving input through pages that would have been recognized as different during non-sharing, a unique page identifier is assigned to each net page. This page ID is accurate enough to distinguish a very large amount of netpages. If the environment is small, it does not require as much accuracy as in a large environment.

Each reference to the page specification is encoded in a printed tag. The tag identifies the unique page where the tag is represented and indirectly identifies the page specification. In a preferred embodiment, the tag also identifies its own location on the page. The characteristics of the tags are described in more detail below.

The tag is printed with infrared-absorptive ink on an infrared-reflective layer, such as conventional paper. Near-infrared wavelengths are invisible to the naked eye but can be easily detected by solid-state image sensors with suitable filters. Sensors that respond to the relevant wavelength (s) can be used, in which case no filter is required. Other wavelengths may also be used if suitable substrates and sensors are used.

The tag is detected by the area image sensor of the netpage pen, and the tag data is sent to the netpage system via the nearest netpage printer. The pen communicates with the netpage printer via a short-range radio link as wireless. The tags are small enough and tightly arranged so that the pen can reliably detect at least one tag even if the page is clicked only once. Since the interaction is stateless, it is important for the pen to detect the page ID and location whenever it interacts with the page. The tags are error-correctably coded to withstand partial surface damage.

The netpage page server maintains a unique page instance for each printed netpage, thereby maintaining a set of distinct values supplied by the user to the input area in the page specification for each printed netpage. To help.

The relationship between page specifications, page instances, and printed netpages is shown in FIG. In a preferred embodiment, the page instance is coupled to both the netpage printer printing the page instance and, if known, the netpage user who requested it. It is not necessary for the page instance to be associated with the netpage printer that printed the corresponding physical page or the netpage user who requested it or the netpage user printed for it.

1.2 NETPAGE TAGS

1.2.1 TAG DATA CONTENT

In a preferred embodiment, each tag identifies the area where the tag is located and the location of the tag within that area. The tag may include a flag associated with the entire area or the tag. For example, one or more flag bits may signal to the tag sensing device without the tag sensing device having to refer to the specification of that region to provide feedback indicative of the action related to the immediate area of the tag. . The Netpage pen may, for example, emit an "active area" LED when in the area of the hyperlink.

As will be described more clearly below, in a preferred embodiment, each tag is an easy to recognize, invariant structure that aids in the initial sensing and helps to minimize the effects of warps due to the surface or sensing process. (easily recognized invariant structure). The tags preferably cover the entire page and are small enough and tightly arranged so that the pen reliably illuminates at least one tag even if the pen makes only one click on the page. Since the interaction is stateless, it is important for the pen to detect the page ID and location whenever it interacts with the page.

In a preferred embodiment, the region referred to by the tag matches the entire page, so that the region ID encoded in the tag is the same as the page ID of the page on which the tag appears. In other embodiments, the region referenced by the tag can be any subregion of a page or other surface. For example, it can match an area of an interactive element, in which case the area ID can directly identify the interactive element.

Each tag usually contains a 16-bit tag ID, at least 90-bit area ID, and a plurality of flag bits. Assuming up to 64 tag densities per square inch, 16-bit tag IDs support up to 1024 square inches. A wider area can be mapped sequentially without simply improving the accuracy of the tag ID by simply adjoining the area and the map. It is easy to distinguish the area ID from the tag ID in most cases. For most purposes, the association of both is considered as a globally unique tag ID. Conversely, it is also convenient to introduce a structure such as defining the x and y coordinate systems of the tag into the tag ID. Domain ID of 90 bits ²⁹⁰ may be such that (i. E. 10 or ²⁷ 1 1 jobae 1000 jobae of) different regions are uniquely identified. The tag may also include type information, and the area may be tagged by mixing tag types. For example, one region may be tagged with one set of tags encoding the x coordinate system and another set of tags encoding the y coordinate system. The accuracy of the area ID and tag ID also depends on the environment in which the system is to be used.

1.2.2 Tag Data Encoding

In one of the embodiments, 120-bit tag data may be redundantly encoded using a (15,5) Reed-Solomon code. This yields 360 encoded bits consisting of six codewords, each with fifteen 4-bit symbols. The (15,5) code can tolerate up to five symbolic errors to be corrected per codeword. That is, up to 33% symbol error can be tolerated for each codeword.

Each 4-bit symbol is represented spatially coherent way within the tag, and the symbols of the six codewords are represented spatially interleaved within the tag. This allows a burst error (an error affecting a plurality of spatially adjacent bits) to damage only the minimum number of symbols and to damage only the minimum number of symbols within any codeword, so that the error can be completely corrected. To ensure the maximum possible potential.

(15,5) Any appropriate error correction code in place of the Reed-Solomon code, for example Reed-Solomon code with some redundancy and the same or different symbols and codewords, Or other kinds of code, convolutional code (e.g., Stephen B. Wicker, Error Control Systems for Digital Communication and Storage, Prentice-Hall 1995, hereby incorporated as cross-reference).

1.2.3 Physical Tag Structure

The physical representation of the tag, as shown in FIG. 5, includes a fixed target structure 15, 16, 17 and a variable data area 18. The fixed target structure allows a sensing device such as a Netpage pen to detect the tag and infer the three-dimensional orientation of the tag in relation to the sensor. The data area contains a representation of each bit of encoded tag data.

For proper reproduction of the tag, the tag is represented with a resolution of 256 x 256 dots. When printed at 1600 dpi, this results in a tag with a diameter of about 4 mm. At this resolution the tag is designed to be surrounded by a "quiet area" having a radius of 16 dots. Invisible areas are also provided by adjacent tags, so only 16 dots are added to the effective diameter of the tag.

The tag contains six target structures. The sense ring 15 allows a sensing device to detect the tag for the first time. The ring is easily detectable because it can eliminate most of the effects of perspective distortion with only a slight aspect ratio correction, which is rotationally invariant. Orientation axis 16 allows the sensing device to determine the approximate two-dimensional orientation of the tag as the sensor shakes. The direction axis is oblique to provide a unique direction. Four perspective targets 17 allow the sensing device to infer the correct two-dimensional perspective transformation of the tag, thus inferring the exact three-dimensional position and relative direction of the tag relative to the sensor.

All target structures become excessively large to improve resistance to noise.

The overall tag is circular in shape. This, among other things, aids in optimal tag packing on irregular triangular grids. In conjunction with the circular sense ring, this allows the circular placement of data bits within the tag to be optimal. To maximize the size, each data bit is represented by a radial wedge in the form of being surrounded by two radial lines and two concentric circular arcs. Each wedge has a minimum dimension of 8 dots at 1600 dpi and is designed such that its base (inner arc) at least corresponds to this minimum size. The radial height of the wedge is always equal to its minimum size. Each 4-bit data symbol is represented by an array of 2x2 wedges. The radial height of the wedge 510 is always equal to the minimum dimension. Each 4-bit data symbol is represented as an array 518 of 2x2 wedges 510, best shown in FIG.

Fifteen four-bit data symbols in each of the six codewords are assigned to four concentric symbol rings 18a-18d in an interleaved fashion. The symbols are alternately arranged to progress in a circle around the tag.

Interleaving is a design to minimize the average spatial distance between any two symbols of the same codeword. Other arrangements of codewords or their data symbols may also be used.

The physical layout of the tags or the shape and / or arrangement of the data symbols within each tag is not essential to the operation of the present invention. It is only necessary that each tag encodes enough information for its intended use. The use of redundancy in the tag is preferred, but at the base level it is not essential to the operation of the present invention. Such other tag arrangements can be used. Examples of other tag structures are described in U.S. Pat. .                 

In order to support a "single-click" interaction with the tagged area via the sensing device, the sensing device may include at least one complete tag within the field of view in any area, in any direction. Should be able to be observed. The required radius of the viewing area of the sensing device is thus a function of the size and spacing of the tags.

Assuming a tag shape that is circular, the minimum radius of the sensing observation area can be obtained when the tag is placed on the equilateral triangle grid as shown in FIG.

1.2.4 Tag Image Processing and Decoding

Tag image processing and decoding is performed by a sensing device such as the Netpage pen shown in FIG. Once the acquired image is obtained from the image sensor, the dynamic range of the image is determined 20. The median of the range is then selected 21 as the binary threshold of the image. The image is then thresholded and segmented 22 into a connected pixel region (ie, shape 23). Shapes too small to represent the tag target structure are discarded. The size and center of each shape are also calculated.

The binary shape moment 25 for each shape is then calculated, thus providing a basis for placing the target structure. The central shape moment is invariant in position due to its nature, and it is easy to make it invariant even if the scale, aspect ratio and rotation change.                 

A ring target structure 15 is hypothetically placed 26. The ring has the advantage that it is most characteristic when perspective-distorted. Matching proceeds by aspect-normalizing and rotation-normalizing the moment of each shape. Once the second-order moment has been normalized, the ring is easy to recognize even if perspective distortion is significant. Both the original appearance and rotation of the ring 27 are useful for estimating a perspective transform.

Next, an axis target structure 16 is disposed 28. Matching proceeds by applying a normalization of the ring to the moment of each shape and rotationally normalizing the resulting moment. Once the secondary moment is normalized, the axis target is easily recognized. Note that one third order moment is required to clarify the two possible directions of the axis. To make this possible, the shape is carefully tilted in one direction. It should also be noted that since perspective distortion can conceal the axis of the axis target, it can only rotationally standardize the axis target after the standardization of the ring is applied. The original rotation of the axial target is useful for estimating the rotation of the tag due to pen shake 29.

Finally, four perspective target structures 17 are arranged (30). The position of the see-through target structure can be well estimated based on the known spatial relationship with the ring and shaft target, the appearance and rotation of the ring, and also the rotation of the shaft. Matching proceeds by applying the standardization of the ring to the moment of each shape. Once the second moment is normalized, the circular perspective target is easily recognized and treated as the match closest to each estimated position. The original center of the three perspective targets then becomes a corner 31 of perspective distorted cells of known size in tag space, with eight-degree-of-freedom perspective transformation 33. Is inferred by solving a well-known equation for the relationship between the order of four tag-space and image-space (Heckbert, P., Fundamentals of Texture Mapping and Image Warping, Masters). Thesis, Dept. of EECS, U. of California at Berkeley, Technical Report No. UCB / CSD 89/516, June 1989, the contents of which are incorporated herein by cross-reference).

Perspective transformation from inferred tag-space to image-space converts the position of the data bits in each known tag space to bilinearly interpolate 36 corresponding four adjacent pixels in the input image. Real-valued positions are used to project 36 into the used image space. The previously calculated image threshold 21 is used to threshold the resulting value to yield a final bit value 37.

Once all 36 data bits 37 are obtained in this manner, each of the six 60-bit Reed-Solomon codewords is decoded to provide 20 decoded bits 39, all 120 Decoded bits are provided. It should be noted that codeword symbols are sampled in the order of codewords, so that codewords are implicitly de-interleaved during the sampling procedure.

As mentioned above, the physical tag structure or encoding system is not essential to the present invention, and other physical arrangements of each tag may also be used. Recognizing and decoding the tag image to recover the encoded data depends on the physical structure of the tag and the system used to encode the data redundantly.

The ring target 15 is searched only in the subregions of the image, which can ensure that the ring associated with the image (if present, is part of a complete tag). If no complete tag is found and successfully decoded, no pen position is recorded for that frame. Given a suitable processing power and ideally a non-minimal field of view 193, an alternative method is to search for another tag in the image.

The obtained tag data indicates the identity of the area, including the tag and the position of the tag within the area. Not only the overall orientation of the pen 35, but also the exact location of the pen nib 35 in that area, is the perspective transformation 33 observed on the tag and the physical axis of the pen and the optical axis of the pen ( can be inferred from known spatial relationships between the optical axes (34).

1.2.5 ALTERNATIVE TAG STRUCTURES

The tag structure just described is designed to allow for both regular tilings of planar surfaces and irregular tilings of nonplanar surfaces. In general, regular tilings are not possible on non-planar surfaces. In the more common case of a planar surface where regular tiling of tags is possible, ie a surface such as a sheet of paper, etc., more effective tag structures can be used, utilizing the regular nature of tiling.

Another tag structure that is more suitable for regular tiling is shown in FIG. 6A. The other tag 4 is square and has four perspective targets 17. It is structurally similar to the tags described by Bennett et al in US Pat. No. 5051746. The tag represents sixty four-bit Reed-Solomon symbols 47 that are a total of 240 bits. The tag is one dot 48, each one bit, and each zero bit if there is no corresponding dot. As shown in Figures 6B and 6C, perspective targets are designed to be shared between adjacent tags. 6B shows a square tiling of 16 tags and a corresponding minimum field of view 193, which should span the diagonal of the two tags. 6C shows square tiling of nine tags and includes all 1 bits for illustrative purposes.

Using the (15,7) Reed-Solomon code, 112 bits of tag data are redundantly encoded to produce 240 encoded bits. Four codewords are spatially interleaved within a tag to maximize resilience to burst errors. Assuming a 16-bit tag ID as before, this allows region IDs of up to 92 bits.                 

The tag's data-bearing dots 48 are designed so that they do not overlap with their neighbors so that groups of tags do not create structures that resemble targets. This also saves ink. Therefore perspective targets allow detection of the tag, so no other targets are needed. Tag image processing proceeds as described in 1.2.4 above, except that steps 26 and 28 are omitted.

Although the tag eliminates the ambiguity of the four possible orientations of the tag for the sensor, it is also possible to embed the orientation data in the tag data. For example, four codewords are shown in FIG. 6D where each symbol is labeled with the number (1-4) of the codeword and the location of the symbol within the codeword (AO). As such, each tag direction may be arranged to include one codeword located in that direction. Tag decoding then consists of decoding one codeword for each direction. Each codeword may include one bit indicating whether it is the first codeword, or two bits indicating which codeword it is. The latter approach has the advantage that, for example, if only one codeword's data content is needed, at most two codewords need to be decoded to obtain the desired data. This is not the case if the region ID is not expected to change during the stroke and is therefore only decoded at the start of the stroke. Then only codewords containing the tag ID are expected during the stroke. Also, since the rotation of the sensing device changes slowly and predictably during the stroke, usually only one codeword per frame needs to be decoded.

It is also possible to go without a perspective target and instead rely on a data representation that is self-resistering. In this case, each bit value (or multi-bit value) is typically represented by a clear glyph. That is, no bit value is represented by the absence of a glyph. This ensures that the data grid is well-populated, so that the grid is reliably confirmed and that perspective distortion is detected and then corrected during data sampling. Allow. In order to allow tag boundaries r to be detected, each tag data must include a marker pattern and these must be redundantly encoded to allow reliable detection. The overhead of such marker patterns is similar to the overhead of explicit perspective targets. One such overview uses dots located at various points with respect to grid vertices to represent different glyphs and thus different multi-bit values. .

1.2.6 Tag Map

When the tag is decrypted, a region ID, a tag ID, and a tag-relative pen transform are output. Before the tag ID and tag-relative pen location are converted to an absolute location within the tagged area, the location of the tag within that area must be known. This is given by a tag map, a function that maps each tag ID to its corresponding position in the tagged area. The tag map class diagram is shown as part of the netpage printer class diagram in FIG.

The tag map represents a scheme used to tag surface areas with tags, and is variable depending on the surface type. If multiple tagged regions share the same tag tiling scheme and the same tag numbering scheme, the tagged regions share the same tag map. It becomes.

The tag map for a region must be retrievable via the region ID. Thus, given any region ID, tag ID, and pen conversion, the tag map can be retrieved, the tag ID can be converted to an absolute tag location within that region, and tag-proportional pen locations (tag-relative pen location) is added to the tag location so that it can provide an absolute pen location within that area.

1.2.7 tagging scheme

As two distinct surface coding schemes, both look at surface coding diagrams that use a tag structure as described earlier in this chapter. The coding table of the embodiment uses the " location-indicating " tag that has already been discussed. Alternatively, an object-indicating tag may be used.

The location-indicating tag includes a tag ID, which, when converted through a tag map associated with a tagged region, provides a unique tag location within that region. The tag-proportional position of the pen is added to this tag position to provide the position of the pen within that area. This is again used to determine the pen's relative position to the user interface element in the page description associated with that area. Not only the user interface element itself is identified, but also its position relative to the user interface element. The location-marking tag will therefore support the capture of an absolute pen path casually within the zone of a particular user interface element.

The object-display tag includes a tag ID that directly identifies a user interface element in the page specification associated with that region. All tags within a section of a user interface element are all identical, making them indistinguishable from each other and identifying that user interface element. The object-marking tag thus does not obtain the absolute path of the pen. However, it can support a relative pen path. As long as the position sampling frequency exceeds two times the encountered tag frequency, the displacement from one sampled pen position to the next can be accurately determined.

Either tag table allows the tag data to be read by the sensing device and the tag is user interactive in that it interacts with the print page using the appropriate sensing device to generate an appropriate response in the netpage system. Like elements, they work in collaboration with related visual elements on the netpage.

1.3 DOCUMENT AND PAGE DESCRIPTIONS

Preferred embodiments of the document and page specification class diagrams are shown in FIGS. 25 and 26.

In a netpage system, documents are described at three levels. At the most abstract level, document 836 has a hierarchical structure whose terminal elements 839 are associated with content objects 840, such as text objects, text style objects, image objects, and the like. ) Once a document is output on a printer according to a particular user's scale factor preference at a particular page size, the document is paginated or otherwise formatted. The formatted terminal element 835 will in some cases be associated with a Content Object that is different from the Content Object that the corresponding Terminal Element is associated with, particularly where the Content Object is style-related. In addition, each printed instance of the document and page is also distinguished to allow input obtained through a particular page instance that is recorded separately from input obtained through other instances of the same page description. Will be explained.

Thanks to the existence of the most abstract document specification on the page server, the user can request a copy of the document without being forced to accept the particular format of the original document. For example, a user may request a copy through a printer with different page sizes. Conversely, thanks to the presence of a formatted document specification on the page server, the page server can also efficiently interpret the user's actions on a particular printed page.

The formatted document 834 consists of a set of formatted page descriptions 5, each page specification consisting of a set of formatted terminal elements 835. Each formatted element has a spatial extent, or zone 58, on the page. This defines the active area of input elements such as hyperlinks and input fields.

Document instance 831 corresponds to formatted document 834. The document instance consists of a set of page instances 830, each page instance corresponding to the page specification 5 of the formatted document. Each page instance 830 describes a single unique printed netpage 1 and records the page ID 50 of that netpage. The page instance is not part of the document instance, even if it represents a copy of the requested page on its own.

The page instance consists of a set of terminal element instances 832. An element instance exists only if it records information specific to that instance. Thus, a hyperlink instance exists for the hyperlink element because it records a transaction ID 55 that is specific for the page instance, and a field instance exists for the page instance. It exists for region elements because it records the input specified. However, element instances do not exist for static elements, such as textflow.

As shown in FIG. 27, the terminal element may be a static element 843, a hyperlink element 844, an area element 845, or a page server command element 846. As shown in FIG. 28, the static element 843 includes a style element 847 with an associated style object 854, a text flow element 848 with an associated text object 855, and an associated image object ( Image element 849 with 856, graphic element 850 with associated graphic object 857, video clip element 851 with associated video clip object 858, associated audio clip object ( audio clip element 852 with an audio clip object 859 or script element 853 with an associated script object 860.

The page instance has a background field 833 which is not used for a particular input element, which is used to record any digital ink obtained on the page.

In an embodiment of the invention, a tag map 811 is associated with each page instance that causes a tag on a page to be translated into a location on that page.

1.4 THE NETPAGE NETWORK

In a preferred embodiment, the netpage network comprises a series of distributed netpage page server 10, netpage registration server 11, and netpage ID server as shown in FIG. The netpage ID server 12, the netpage application server 14, and the netpage printer 601 are connected through a network 19 such as the Internet.

The netpage registration server 11 is a server that permits various network activities by recording a relationship between a user, a pen, a printer, an application, and a publication. The netpage registration server authenticates the user and acts as a signing proxy for the authenticated user in application processing. The netpage registration server also provides a handwriting recognition service. As mentioned above, the netpage page server 10 maintains persistent information about the page description and page instance. The netpage network includes any number of page servers, each of which is responsible for a subset of the page instances. The page server also maintains a user input value for each page instance, so a client, such as a netpage printer, sends the netpage input directly to the appropriate page server. The page server interprets any such input with respect to the corresponding page.                 

The netpage ID server 12 assigns a document ID 51 upon request and provides load-balancing of the page server via its ID allocation scheme. .

The netpage printer uses the Internet Distributed Name System (DNS) or the like to translate the netpage page ID 50 into the network address of the netpage page server that handles the corresponding page instance. .

The netpage application server 13 is a server that runs an interactive netpage application. The netpage publishing server 14 is an application server for publishing netpage documents to a netpage printer. These are described in more detail in section 2.

Netpage servers can run on a variety of network server platforms from manufacturers such as IBM, Hewlett-Packard and Sun. Multiple netpage servers may run concurrently on one host, and one server may be distributed across multiple hosts. Some or all of the functionality provided by the netpage server, in particular the functionality provided by the ID server and page server, may be provided directly by a netpage device, such as a netpage printer on a computer workstation or local network.

1.5 THE NETPAGE PRINTER

The netpage printer 601 is a device registered in the netpage system and prints a netpage document according to a request and a subscription. Each printer has a unique printer ID 62 and is connected to a netpage network via a network such as the Internet, ideally via a broadband connection.

Apart from identification and security settings in non-volatile memory, netpage printers do not include persistent storage. As far as the user is concerned, "the network is a computer". Netpages work in both directions over time and space, driven by a distributed netpage page server 10, independent of a particular netpage printer.

The netpage printer receives a subscribed netpage document from the netpage publishing server 14. Each document is divided into two parts: the page layout and the actual text and image objects that occupy the page. Due to personalization, page arrangements are usually specialized for a particular subscriber, as are pointcasts to that subscriber's printer through the appropriate page server. Text and image objects, on the other hand, are usually shared with other subscribers, as are multicasts to all subscribers' printers through the appropriate page server.

The Netpage Publishing Server optimizes the splitting of document content into pointcasts and multicasts. After receiving the pointcast of the page arrangement of the document, the printer knows which multicasts (if any) should be received.

Once the printer receives the complete page arrangement and the object defining the document to be printed, the printer prints the document.                 

The printer rasterizes and prints odd and even pages on both sides of the paper at the same time. Print engines using a duplexed print engine controller 760 and a Memjet ™ print head 350 are included for this purpose.

The printing process consists of two separate steps: rasterization of the page specification, enlargement and printing of the page image. A raster image processor (RIP) consists of one or more standard DSPs 757 operating in parallel. The dual print engine control unit may be configured as an on-demand processor that performs the enlargement, dithering, and printing of page images in real time in synchronization with the operation of the print head of the print engine.

Printers that are not capable of IR printing have the option to print the tag using IR-absorptive black ink, even if the tag would otherwise limit the area of the page to be blank. have. Such pages will have more limited functionality than IR-printed pages, but are still classified as netpages.

Typical netpage printers print netpages on paper. More specialized netpage printers can print netpages on more specialized surfaces, such as globes. Each printer supports at least one surface type and at least one tag tilting scheme, thus supporting a tag map for each surface type. In practice, the tag map 811, which describes the tag gradient table used to print the document, allows the tags of the document to be correctly interpreted in relation to the document.

Fig. 2 shows a netpage printer class diagram reflecting printer-related information maintained by the registration server 11 on the netpage network.

A preferred embodiment of a netpage printer is described in more detail with reference to FIGS. 11-16 in section 6 below.

1.5.1 Memjet (TM) Printheads

Netpage systems can be operated using printers with a wide range of digital printing techniques, including thermal inkjet, piezoelectric inkjet, laser electrophotographic, and the like. However, in order to accommodate a wide range of users, it is desirable for a netpage printer to have the following characteristics.

Photo quality color printing

High quality text printing

High reliability

· Low printing cost

· Low ink cost

· Low paper cost

Simple operation                 

Extremely quiet printing

Fast print speed

2-sided simultaneous printing

Compact form factor

· Low power consumption

No commercially available printing technique satisfies all of these features.

In order to realize the manufacture of a printer having such a feature, the applicant has invented a new printing technology called Memjet (TM) technology. Memjet ™ is a drop-on-demand inkjet technology with a pagewidth printhead manufactured using microelectromechanical systems (MEMS) technology. . FIG. 17 shows a single print element 300 of a Memjet (trademark) print head. The Netpage wallprinter incorporates 168960 printing elements 300 to implement a 1600 dpi pagewidth duplex printer. The printer prints cyan, magenta, yellow, black and paper conditioner and ink fixative as well as infrared ink at the same time.

The printing element 300 has a length of approximately 110 microns and a width of 32 microns. This array of printing elements is formed on a silicon substrate 301 that incorporates CMOS logic, data transfer, timing, and drive circuitry (not shown).

The main elements of the printing element 300 are a nozzle 302, a nozzle rim 303, a nozzle chamber 304, a fluid seal 305, an ink channel rim (ink channel rim) 306, lever arm 307, active actuator beam pair 308, passive actuator beam pair 309, active actuator anchor (active actuator anchor) 310, manual actuator anchor 311 and ink inlet (312).

The active actuator beam pair 308 is mechanically coupled with the passive actuator beam pair 309 at the junction 319. Both beam pairs are fixed at respective anchor points 310, 311. The combination of elements 308, 309, 310, 311 and 319 form a cantilevered electrothermal bend actuator 320.

18 is a portion of an arrangement of printing elements 300, including a cross section 315 of printing elements 300. In order to clearly express the ink injection hole 312 penetrating the silicon wafer 301, the cross section 315 is shown without ink.

19 (a), 19 (b) and 19 (c) show the operating cycles of the Memjet (trademark) printing element 300.

Fig. 19A shows the stop position of the ink meniscus 316 before printing fine ink droplets. Ink is maintained in the nozzle chamber by the surface tension of the ink meniscus 316 and the fluid chamber 305 formed between the nozzle chamber 304 and the ink channel rim 306.

During printing, the CMOS circuitry of the print head distributes data from the print engine to the appropriate print elements, latches the data, and buffers the data to drive the electrodes 318 of the active actuator beam pair 308. This causes Joule heating by causing current to flow about 1 microsecond through the beam pair 308. Beam pair 308 expands due to temperature rise due to Joule heat. Since the passive actuator beam pair 309 is not heated, it will not expand, resulting in a stress difference between the two beam pairs. This stress difference is partially resolved by the cantilever end 320 of the hot electron bending actuator bending towards the substrate 301. The lever arm 307 transmits this movement to the nozzle chamber 304. The nozzle chamber 304 moves about 2 microns to a position as shown in FIG. 19 (b). This increases the ink input, expanding the ink meniscus 316 while forcing the ink 321 at the nozzle 302. The nozzle rim 303 prevents the ink meniscus 316 from spreading to the surface of the nozzle chamber 304.

When the temperatures of the beam pairs 308 and 309 are equal, the actuator 320 returns to its original position. This helps stop the ink droplets 317 from the ink 312 in the nozzle chamber, as shown in Fig. 19 (c). The nozzle chamber is refilled by the surface tension at meniscus 316.

20 is a fragment of the print head 350. In a netpage printer, the length of the print head is the full width of the transverse direction 351 (usually 210 mm). The length of the fragment shown is 0.4 mm (about 0.2% of a complete print head). In printing, the paper passes the fixed print head in the longitudinal direction 352. The print head has six rows of interlocking printing elements 300 for printing six colors or types of ink supplied from the ink inlets 312.

In order to protect the weak surface of the print head during operation, a nozzle guard wafer 330 is attached to the printhead substrate 301. Each nozzle 302 has a nozzle guard hole 331 corresponding to the nozzle on which ink droplets are burned. In order to prevent the nozzle protection hole 331 from being clogged by fibrous or other debris of paper, the filtered air is extruded into the air inlet 332 and the nozzle protection hole during printing. In order to prevent the ink 321 from drying out, the nozzle guard is sealed while the printer is waiting.

1.6 The Netpage Pen

The active sensing device of the netpage system is usually a pen 101 capable of obtaining and decoding an IR position tag from a page through an image sensor by using an embedded controller 134. . An image sensor is a solid state device in which an appropriate filter is provided to detect only near-infrared wavelengths. As described in more detail below, the system can detect when the pen tip is in contact with a surface, and the pen is tagged with a degree sufficient to obtain handwriting (i.e., resolution of 200 dpi or higher at frequencies above 100 Hz). Can be detected. The information obtained by the pen is encoded and wirelessly transmitted to a printer (or base station), where the printer or base station interprets the data in relation to the (known) page structure, or in one of the embodiments, Information is sent to the netpage server.

In a preferred embodiment of the netpage pen, the pen performs the operations of the general marking ink pen and the non-marking stylus together. However, for display purposes, it is not necessary to use a netpage system as a browsing system as used as an internet interface. Each netpage pen is registered with the netpage system and has a unique pen ID 61. FIG. 23 is a netpage pen class diagram showing pen-related information maintained by a registration server 11 on a netpage network.

When the nib contacts the netpage, the pen determines the relative position and orientation of the pen relative to the page. A force sensor is attached to the pen tip, and the pressure applied to the pen tip is interpreted based on a threshold to indicate whether the pen is "up" or "down". This allows interactive elements on the page to be 'clicked' by pressing the pen tip to request information from the so-called network. Furthermore, the pressure is obtained as a continuous value to account for so-called full dynamics for verifying the signature. The pen nib can move when a certain pressure is applied that is greater than normally applied during writing. To “click” the user applies enough pressure to move the pen tip. This may provide more desirable feedback compared to the feedback provided by the non-moving nib.

The pen determines the position and orientation of the pen tip on the netpage by imaging the nib adjacent area 193 of the page in the infrared spectrum. It decodes the closest tag and calculates the relative position of the pen tip relative to the tag from the perspective distortion observed in the illuminated tag and the known geometry of the pen optics. Although the position resolution of a tag may be low because the density of the tag on the page is inversely proportional to the size of the tag, the adjusted position resolution does not provide the minimum resolution needed to accurately read handwriting. It's quite high enough to go over.

The pen's relative motion to the netpage is obtained as a series of strokes. One stroke consists of a series of pen positions on a time-stamped page that begin at pen-down occurrences and end at pen-up occurrences. Each time the page ID is changed, the stroke is appended with the page ID 50 of the netpage. The page ID is at the beginning of the stroke in the general environment.

Each netpage pen has a current selection 826 associated with it, allowing the user to perform operations such as copying and pasting. The selection records the time for the system to discard after a defined time period has elapsed. The current selection describes the area of the page instance. It consists of the newest digital ink stroke in relation to the background area of the page, obtained through the pen. Once submitted to an application via the selected hyperlink activation, it is interpreted in an application-specific manner.

Each pen has a current nib 824. This is the pen tip last known to the system by the pen. In the case of the default netpage pen as described above, the display black ink nib or the non-display stylus nib becomes the current nib. Each pen also has a current nib style 825. This is the last nib style associated with the pen by the application, such as in the case of a response to the user selecting a color from a palette. The default nib style is the nib style associated with the current nib. The stroke obtained through the pen is appended with the current nib style. As strokes continue to be reproduced, they are duplicated in the nib style attached to them.

Whenever the pen is within range of the printer it is communicating with, the pen slowly flashes the "online" LED. If the pen fails to decode a stroke on a page, the pen momentarily activates an "error" LED. If the pen succeeds in decrypting the stroke on the page, the pen momentarily activates the "ok" LED.

The series of acquired strokes is called digital ink. Digital ink forms the basis for digital exchange of drawings and handwriting, online recognition of handwriting, and online verification of signatures.

The pen is wireless and sends digital ink to a netpage printer via a short-range wireless connection. Transmitted digital ink is encrypted for privacy and security and packetized for efficient transmission, but is always flushed at the time of pen-up to ensure timely handling at the printer. do.

When the pen is out of range of the printer, it buffers the digital ink in internal memory, which has a continuous handwriting capacity of 10 minutes or more. If the pen falls back within range of the printer, it transfers the buffered digital ink.

A pen can be registered with any number of printers, but since all state data is present on both netpages on paper and netpages on the network, it is most likely which pen is communicating with which printer at a particular time. It doesn't matter.

Preferred embodiments of the pen are described in more detail with reference to FIGS. 8 to 10 in section 6 below.

1.7 NETPAGE INTERACTION

The netpage printer 601 receives data relating to strokes from the pen 101 when the pen is used for interaction with the netpage 1. The encoded data 3 of the tag 4 is read by the pen when the pen is used to perform an operation such as a stroke. The data allows the identity of the particular page and its associated interactive elements to be determined and an indication of the position of the pen relative to the page. Indication data is sent to the printer, where the indication data is via the DNS network of the netpage page server 10 that maintains the page ID 50 of the stroke to which the page instance 830 corresponds. Convert to an address. The printer then sends the stroke to the page server. If the page was recently identified in the previous stroke, the printer may already have the address of the relevant page server in the cache. Each netpage consists of a compact page layout maintained continuously by a netpage page server (see below). Page layout is usually associated with objects such as images, fonts, and text that are stored somewhere in the netpage network.

When the page server receives the stroke from the pen, the page server retrieves the page specification that applies to the stroke and determines which elements of the page specification cross the stroke. The page server can then interpret the stroke in the context of the form of the associated element.

A "click" is a stroke in which the distance and time between the pen down position and the subsequent pen up position are both less than some small maximum. Objects activated by a click usually require a click to be activated, so longer strokes are ignored. Failure of a pen action to register, such as "sloppy click", is indicated by no response from the pen's "ok" LED. However, in the portion where the netpage includes the "click" button, it can be registered when both the pen down and pen up positions are within the area of the button.

There are two input elements in the netpage page specification: hyperlinks and form fields. Input through the form region can also trigger activation of the associated hyperlink.

1.7.1 Hyperlinks

Hyperlinks are a means of sending a message to a remote application, usually driving a printed response in a netpage system.

The hyperlink element 844 includes an application 71 that handles activation of the hyperlink, a link ID 54 that identifies the hyperlink for the application, and a user's application for hyperlink activation. An alias used to request the system to include an alias (65) and a description that is used when a hyperlink is recorded as a favorite or appears in the user's history. ). A hyperlink element class diagram is shown in FIG. 29.

When the hyperlink is activated, the page server sends a request to an application somewhere on the network. The application is identified by an application ID 64, which is converted in the usual way via DNS. There are three kinds of hyperlinks: generic hyperlinks 863, form hyperlinks 865, and optional hyperlinks 864, as shown in FIG. A general hyperlink may execute a request for a linked document or simply signal a preference to a server. The form hyperlink submits the corresponding form to the application. The selection hyperlink submits the current selection to the application. If the current selection includes, for example, a single-word of text, the application may be in a single-page document or other language that gives meaning to that word in the context in which it appears. Can return a translation of. Each type of hyperlink is characterized by what information is submitted to the application.

The corresponding hyperlink instance 862 records the process ID 55 which can be specified according to the page instance on which the hyperlink instance appears. The process ID may identify, as an application, data specific to the user, for example, a "shopping cart" under purchase, which is held by a purchasing application on the user side.

The system includes the pen's current selection 826 at selection hyperlink activation. If the form hyperlink has a "submit delta" attribute set, the system will add the content of the relevant form instance 868 to form hyperlink activation, even if only the input after form submission is included. Include. The system includes an efficient return path for all hyperlink activations.

Hyperlinked group 866 is a group element 838 having an associated hyperlink as shown in FIG. If the input occurs through any area element in the group, the hyperlink 844 associated with that group is activated. Hyperlinked groups can be used to associate hyperlink behavior with areas such as checkboxes. It can also be used to provide continuous input to an application in conjunction with the "Submit Delta" attribute of a form hyperlink. Thus, it can be used to support a "blackboard" interactive model, i.e., an interactive model that is shared as input is obtained as soon as an input occurs.

1.7.2 Forms

The form defines a collection of related input areas that are used to obtain related sets of inputs through a printed netpage. The form allows the user to submit one or more parameters to an application software program running on the server.

Form 867 is a group element 838 in the document hierarchy. The form ultimately contains a set of terminal field elements 839. Form instance 868 represents a printed instance of the form. The form instance consists of a set of region instances 870 corresponding to the region elements 845 of the form. Each region instance has an associated value 871, whose type depends on the type of the corresponding region element. Each field value records its input through a particular printed form instance, ie, through one or more printed netpages. The form class diagram is shown in FIG.

Each form instance has a status 872 indicating that the form is active, frozen, submitted, invalid, or expired. The form is activated when it is first printed. The form is frozen once it is signed. The form is submitted once one of its submitting hyperlinks is active, unless the hyperlink has a "submit delta" attribute set. If the user calls the invalid form to reset the form or copies the form page command, the form becomes invalid. The form expires when the time that the form is active has passed a certain lifetime. Form input is allowed while the form is active. Input through non-activated forms is obtained instead from the base area of the associated page instance.

If the form is active or frozen, submission of the form is allowed. If the form is not active or frozen, any form submission attempt will be rejected and a form status report will be derived instead.

Each form instance provides a version history with respect to any form instance originating therefrom (59). This allows all versions of the form other than the last version to be excluded from search in a particular time period.

All input is obtained as digital ink. Digital ink 873 consists of a timestamped stroke group 876, with each time-recorded stroke group consisting of a set of styled strokes 875. Is made of. Each stroke consists of a pen position 876 in which a set of times are recorded, and each time recorded pen position also includes pen orientation and nib force. . A digital ink class diagram is shown in FIG.

The region element 845 can be a checkbox region 877, a text region 878, a pictorial region 879, or a signature region 880. The region element class diagram is shown in FIG. The digital ink obtained in the area 58 of one area is designated as that area.

The checkbox area has an associated boolean value 881 as shown in FIG. A mark (point, scissors, line drawing, zigzag, etc.) is obtained in the area of the check box area so that the true value of that area is assigned.

The text area has an associated text value 882 as shown in FIG. The digital ink obtained in the area of the text area is automatically converted into text through online handwriting recognition, and the text is designated as the field's value of the area. Online handwriting recognition is known (eg Tappert, C., CYSuen and T. Wakahara, "The State of the Art in On-Line Handwriting Recognition", IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 12 , No. 8, August 1990. The contents are incorporated herein as cross-references). The specialized text area contains date and numeric fields.

The signature area has an associated digital signature value 883 as shown in FIG. When digital ink is acquired in an area of the signature area, the relationship with the identity of the owner of the pen is automatically verified, and a digital signature is created for the content of the form that forms part of that area and assigned to the area value of that area. do. The digital signature is generated using the pen user's private signature key specific to the application that owns the form. Online signature verification is known (e.g., Plamondon, R. and G. Lorette, "Automatic Signature Verification and Writer Identification-The State of the Art", Pattern Recognition, Vol. 22, No. 2, 1989. The content is incorporated herein by cross-reference).

Area elements are concealed if their "hidden" attribute is set. The concealed region element does not have an input region on the page and does not accept input. It may have an associated region value that is included in the form data when the form containing that region is submitted. "Editing" commands, such as strike-through, that indicate deletion, can also be recognized in the form area.

Handwriting recognition algorithms are more effective at 'online' (accessing the dynamics of pen movements) than 'offline' (accessing only bitmaps marked with pens), so that discretely written characters On discretely-written characters can be recognized with relatively high accuracy without the need for a writer-dependent training step. Author-dependent handwriting models are created automatically over time, but they can be created earlier if needed.

As mentioned above, the digital ink consists of a series of strokes. Any stroke starting at the zone of a particular element is attached to the digital ink stream of that element and ready for interpretation. Strokes that are not attached to the digital ink stream of the object are attached to the digital ink stream of the background area.                 

Digital ink obtained in the background area is interpreted as a selection pseudo indication. Although the actual interpretation is application-specific, the circumscription of one or more objects is generally interpreted as selecting a bounded object.

Table 2 summarizes the pen's various interactions with these netpages.

Table 2-Summary of Pen and Netpage Interactions

Object form Pen typing action Hyperlink Normal click Submit Action to Application form click Submit Form to Application Selection click Selective submission to the application Form area Checkbox Random display Specify true value for region text Handwriting Converting digital ink to text; Assign text to an area Drawing Digital ink Assign digital ink to an area signature signature Digital ink signature verification; Digital signature generation of forms; Assign digital signatures to zones none - Limit specification Assign digital ink to the current selection

The system maintains a current selection for each pen. The selection simply consists of the newest stroke obtained in the background area. The selection is cleared after an inactivity timeout to ensure predictable behavior.

Raw digital ink obtained in all areas is maintained on the netpage server and optionally sent along with the form data when the form is submitted to the application. This allows the application to interrogate raw digital ink in case of original conversion, such as conversion of handwritten text. For example, human intervention at the application level for modalities that fail a consistency check specific to a given application. As an extension to this, the entire base area of the form may be designated as a drawing field. Based on the presence of digital ink outside the explicit area of the form, the application routes the form to a human operator, assuming that the user can direct modifications to the filled area outside of the area. can do.

38 is a flowchart of a procedure for handling pen input relating to netpages. This procedure includes receiving a stroke from a pen (884); Step 885, at page 8, that page ID 50 identifies the associated page instance 830; Retrieving page specification 5 (886); Identifying 887 the formatted element 839 of the region 58 where the strokes intersect; Determining 888 whether the formatted element corresponds to an area element, and if so attaching the received stroke to the digital ink of area value 871, accumulating the digital ink of that area; Interpreting 833, and determining 894 if the region is part of a hyperlinked group 866, and if so, activating the associated hyperlink 895; Otherwise, determining 889 if the formatted element corresponds to a hyperlink element, and if so, activating 895 the corresponding hyperlink; Otherwise in the absence of the input area or hyperlink, attaching the received stroke to the digital ink in the background area 833 (890); And copying the received stroke to the current selection 826 of the current pen, as maintained by the registration server, 891.

Fig. 38A is a detailed flowchart of step 893 of the procedure shown in Fig. 38, in which the accumulated digital ink of the area is interpreted according to the type of the area. The procedure includes determining whether the region is a check box (896) and determining whether the digital ink represents a check mark (897), and if so, specifying a true value of true as the region value (898). ; Otherwise, determining (899) whether or not the area is a text area; if so, converting digital ink to computer text (900) by means of a suitable registration server; and designating the converted computer text as an area value ( 901); Otherwise, determining (902) whether the area is a signed area; if so, authenticating the digital ink as a signature of the owner of the pen (903) by means of an appropriate registration server and by a registration server to the corresponding application. Generating a digital signature for the content of the corresponding form using the associated pen owner's secret signature key (904) and designating the digital signature as an area value (905).

1.7.3 Page Server Commands

Page server commands are commands that are handled locally by the page server. Page server commands work directly with forms, pages, and document instances.

The page server command 907 is a void form command 908, a duplicate form command 909, a reset form command 910, a get form status command as shown in FIG. 911, duplicate page command 912, reset page command 913, page status get command 914, duplicate document command 915, reset document command 916, or document status get command ( 917).

Invalid form command invalidates the corresponding form instance. The duplicate form command invalidates the corresponding form instance and creates an active printed copy of the current form instance with the preserved area value. The copy contains the same hyperlink transaction ID as the original, so it is indistinguishable from the original in the application. The reset form command invalidates the corresponding form instance and creates an active print copy of the form instance with the discarded build value. The Get Form Status command generates a printed report on the status of the corresponding form instance, including who published it, when it was printed, to whom it was printed, and the status of the form instance's form.

Since the form hyperlink instance contains a processing ID, the application must include the creation of a new form instance. Therefore, buttons requesting new form instances are usually implemented as hyperlinks.

The duplicate page command creates a print copy of the corresponding page instance with the preserved background area value. If the page contains a form or is part of a form, the duplicate page command is interpreted as a duplicate form command. The reset page command creates a print copy of the corresponding page instance with the discarded background area value. If the page contains a form or is part of a form, the reset page command is interpreted to be a reset form command. The Get Page Status command generates a printed report about the status of the corresponding page instance, including who published it, when it was printed, to whom it was printed, and any form of status that includes or is part of.

The netpage logo that appears on all netpages is usually associated with duplicate page elements.

If a page instance overlaps with field values preserved, the field values are printed in their native form. That is, the checkmark appears as a standard checkmark graphic, and the text appears as typeset text. Only drawings and signatures appear in their original form, with signatures accompanied by standard graphics indicating successful signature verification.

The duplicate document command creates a print copy of the corresponding document instance with the preserved background area value. If the document contains some form, the duplicate document command copies the form in the same way as the duplicate form command. The reset document command creates a print copy of the corresponding document instance with the discarded background area value. If the document contains some form, the reset document command resets the form in the same way as the reset form command. The Get Document Status command generates a printed report on the status of the corresponding document instance, including who published it, when it was printed, to whom it was printed, and the status of the form included.

If the "on selected" attribute of a page server command is set, the command is performed on the page identified by the pen's current selection rather than on the page containing the command. This causes a menu of page server commands to be printed. If the target page does not contain a page server command element for the specified page server command, the command is ignored.

An application can provide application-specific handling by embedding related page server command elements into hyperlinked groups. The page server activates hyperlinks related to hyperlinked groups rather than performing page server commands.

If the "hidden" attribute is set, the page server command element is hidden. The hidden command element does not have an input area on the page and thus cannot be activated directly by the user. However, if the page server command has an "on selected" attribute set, it can be activated through a page server command embedded in another page.

1.8 STANDARD FEATURES OF NETPAGES

In a preferred embodiment, each netpage is printed on the underside with a netpage logo indicating that it is a netpage, thus having interactive properties. The logo also acts as a copy button. In most cases, pressing a logo creates a copy of the page. For forms, the button creates a copy of the entire form. And, in the case of a secure document such as a ticket or coupon, the button pulls out a description or advertisement page.

The default single-page copy function is handled directly by the relevant netpage page server. Special copy functions are handled by linking the logo button to the application.

1.9 USER HELP SYSTEM

In a preferred embodiment, the netpage printer has one button labeled " Help. &Quot; When the button is pressed, the button pulls out a page of information containing the following:

· Printer Connection Status

· Printer Supplies Status

Top-level help menu

Document function menu

· Top level netpage network directory

The Help menu provides a hierarchical manual on how to use the netpage system.

The document function menu includes the following functions.

· Print a copy of the document

· Print a clean copy of the form

· Print document status

The document function is initiated by simply pressing a button and touching any page of the document. The status of the document indicates who, when it was published, to whom it was delivered, and to whom later and as a form.

The netpage network directory allows users to navigate the hierarchy of publications and services on the network. Alternatively, a user may call the "yellow page" of the Netpage network "900" to speak with a human operator. The agent can locate the desired document and route it to the user's printer. Depending on the type of document, the publisher or user pays some "phone book" service fees.

It is clear that the help page is not available if the printer cannot print. In this case the "error" light is lit and the user can request a remote diagnosis on the network.

2. Personalized Publication Model

In the description below, news is used as a canonical publication example to describe the personalization mechanism of a netpage system. Although news is often used in the limited sense of news from newspapers and newsmagazines, the intended scope in this context is much broader.

In a netpage system, the editorial content and advertising content of a news publication are personalized using different mechanisms. The edits are personalized according to the interest profile explicitly specified by the reader and the interest profile implicitly implied. The advertising content is personalized according to the reader's locality and demographic.                 

2.1 EDITORIAL PERSONALIZATION

Subscribers can use two news sources: one that delivers news publications, and one that delivers news streams. While news publications are collected and edited by publishers, news streams are collected by news publishers or by specialized news aggregators. News publications usually correspond to traditional newspapers and magazines, while news streams can include "raw" news, comic strips, freelance writer's columns, friends' bulletin boards, or reader's e-mails from news services. It can be varied.

The netpage publishing server supports the aggregation of multiple news streams as well as the publication of edited news publications. By dealing with the aggregation of the news stream directly selected by the reader and the resulting formatting, the server can place an advertisement on a page that would otherwise have no editorial control.

The subscriber builds a daily newspaper by selecting one or more provided news publications and generating each personalized version. The resulting daily editions are printed and bound together in one newspaper. Various members of the generation usually express their different interests and preferences by selecting and customizing different daily publications.

For each publication, the reader can optionally take a specific section. Some sections are daily, while others are weekly. For example, the daily sections available at The New York Times online include "Page One Plus," "National," "International," and "Opinion." Opinion ”,“ Business ”,“ Arts / Living ”,“ Technology ”and“ Sports ”. A suite of available sections, like the default sebset, is specific to the publication.

The reader can expand the daily newspaper by creating a custom section that uses any number of news streams. Custom sections are for e-mail and for notifying friends ("Personal" section), or for monitoring news provided about a specific topic ("Alert" or "Clipping" section). Can be generated.

For each section, the reader can specify its size either quantitatively (e.g., short, medium, or long) or numerically (i.e., by limiting the number of pages), and by The ratio can be specified either quantitatively (eg, high, medium, low, none) or numerically (ie, as a percentage).

The reader also optionally expresses preferences for many short articles or a few long articles. Each article is ideally written (or edited) in both short and long form to support this preference.

Articles can also be written (or edited) in different versions, eg, children's and adult versions, to suit the reader's expected sophistication. The appropriate version is chosen according to the age of the reader. The reader can specify a "reading age" that overrides his biological age.

The articles that make up each section are cooked and prioritized by the editor, and each article is assigned a useful lifetime. By default, articles are delivered to all relevant subscribers in priority order according to the subscriber's space constraint.

In the appropriate section, the reader can optionally enable collaborative filtering. This applies to articles with a sufficiently long expiration date. Each article that satisfies the joint filtering is printed with a rating button at the end of the article. The button can provide simple choices (eg, "good" and "dislike") so that rating the article is not annoying to the reader.

Therefore, articles with high priority and short expiration date are effectively considered by the editor as essential reading and delivered to most subscribers.

Optionally, the reader can specify the serendipity factor quantitatively (eg, to amaze or not amaze me). High windfall factors lower the threshold used for matching during joint filtering. A high factor allows the corresponding section to increase the likelihood that the reader will meet a particular capacity. Different windfall elements can be specified for different days of the week.                 

You can also optionally specify a topic of particular interest, which changes the priority assigned by the editor.

The speed of the reader's internet connection affects the quality of the images being delivered. The reader can optionally specify a preference for a small image, a small image, or both. If the number or size of images is not reduced, the images can be delivered at a lower quality (eg, at a lower resolution or at a higher compression rate). Or all three of the quantity, size, and quality of the image to be delivered may be adjusted.

At the global level, the reader can specify the localization of quantity, date, time and monetary value. It determines whether the unit follows the British metrology or metric system, whether it is translated or annotated for the local timezone and time format, local currency and localization. It includes what is specified. These preferences are basically derived from the reader's location.

To alleviate the difficulty of deciphering due to poor eyesight, the reader can optionally specify a global preference for large presentations. As a result, both text and images are scaled, and each page contains a small amount of information.

The language in which the news publication is published and the resulting text encoding are characteristics of the publication and not the choice of the user. However, the netpage system can be configured to provide automatic translation services in various forms.                 

2.2 ADVERTISING LOCALIZATION AND TARGETING

Since advertising is usually in the context of editing, personalization of edits directly affects advertising content. For example, travel advertisements are more likely to appear in the travel section than elsewhere. The value of editorial content to advertisers (and publishers) depends on their ability to attract multiple readers with adequate demographics.

Effective advertising is based on locality and personal information. Locations address accessibility and specific concerns to specific services, retail outlets, etc., and are relevant to the community and the environment. Biographical information is expected to cover general interests and prejudices as well as likely spending patterns.

A news publisher's most profitable product is a multi-dimensional entity, determined by the geographical coverage of the publication, the size of the readership, the identity of the readership, and the area of the page where the advertisements can be placed. , Which is an advertisement "space".

In the Netpage system, the Netpage Publishing Server sells publications by section, taking into account the geographical coverage of the publication, the readership of the section, the size of each reader's section format, the proportion of each reader's advertisement, and the personal information of each reader Compute roughly the multidimensional scale of the ad space where possible.

Compared to other media, the netpage system allows the advertising space to be defined in greater detail and can be sold in smaller units. Thus, the advertising space can be sold closer to its true value.                 

For example, the same ad "slot," a variable share, allowing each reader's page to randomly receive ads from one advertiser or another while maintaining the overall share of space sold to each advertiser. may be sold to multiple advertisers as a varying proportion.

Netpage systems can link advertisements directly to product details and online purchases. Therefore, it increases the intrinsic value of the advertising space.

Since personalization and localization are handled automatically by the netpage publishing server, an advertising aggregator can provide arbitrarily broad coverage in both local and personal information. Subsequent disaggregation is also automatic and efficient. This makes it more cost-effective for publishers to deal with ad recruiters rather than directly recruiting ads. Although ad recruiters take a certain percentage of ad revenue, publishers will find that such changes are profit-neutral because of the greater efficiency of recruitment. Advertisers can act as intermediaries between advertisers and publishers, and can load the same advertisement in multiple publications.

It is worth noting that, due to the more complex advertising space, placing advertisements in netpage publications can be more complicated than placing advertisements in traditional publications. While neglecting the complexity of negotiation between advertisers, ad recruiters, and publishers, a preferred embodiment of the netpage system provides automated support for negotiation, including automated auction support of ad space. to provide. Automation is particularly desirable to attract ads that generate only a small amount of revenue, such as small advertising or extremely local advertising.

Once the advertisement is negotiated, the recruiter receives the advertisement, edits it, and writes it on a netpage ad server. In response, the publisher records the attract of the advertisement on the associated netpage publishing server. When the netpage publishing server arranges each user's personalized publication, the relevant advertisement is selected from the netpage advertisement server.

2.3 USER PROFILES

2.3.1 Information Filtering

Personalization of news and other publications relies on a classification of profile information specific to the user, including:

Publication customizations

Collaborative filtering vectors

Contact details

Presentation preferences

Since the customization of a publication is usually unique to the publication, the customization information is maintained by the associated netpage publishing server.

The joint filtering vector consists of a user's rating of multiple news items. It is used to correlate interests of different users for the purpose of recommendation. There are advantages to maintaining a single joint filtering vector for a particular publication independently, but there are two reasons that it is more realistic to maintain separate vectors for each publication. That is, there may be more overlap between vectors of subscribers to the same publication than between vectors of subscribers to different publications. In addition, the publication will want to display the user's joint filtering vector as part of the brand value of the publication not found elsewhere. Thus the joint filtering vector is also maintained by the associated netpage publishing server.

Connection details, including name, address, postal code, state, country and telephone number, are inherently global and maintained by a netpage registration server.

Display preferences, including quantity, date, and time, are likewise global and remain the same.

Localization of ads depends on the locality that appears in the user's connection details, while targeting of ads is qualitative, such as birth date, gender, marital status, income, occupation, education or age range, and income range. Depends on personal information such as qualitative derivatives

For a user who decides to release personal information for advertisement, the information is maintained by the relevant netpage registration server. In the absence of such information, the advertisement may be targeted based on personal information associated with the user's ZIP or ZIP + 4 code.                 

As shown in Figs. 21, 22, 23 and 24, each user, pen, printer, application provider and application has its own unique identifier, and the Netpage registration server is assigned between them. Maintain a relationship. For the purpose of registration, the publisher is a special kind of application provider, and the publication is a special kind of application.

Each user 800 may be authorized to use a certain number of printers 802, and each printer may be allowed to be available to a given number of users. Each user has a single default printer to which periodical publications are delivered by default (66), and pages printed upon request are delivered through a printer with which the user interacts. The server keeps track of the publishers that authorize it to print to the user's default printer. The publisher does not record the ID of a particular printer, but instead analyzes the ID if necessary.

When the user subscribes to the publication 807 (808), the publisher 806 (i.e., application provider 803) authorizes printing to a particular printer or to the user's default printer. This authorization can be revoked at any time by the user. Each user can have multiple pens 801, but one pen is specific to a single user. If a user is authorized to use a particular printer, that printer can recognize any of the user's pens.

The pen ID is used to find the corresponding user profile maintained by a particular netpage registration server via DNS in a general manner.                 

Web terminal 809 may be authorized to print on a particular netpage printer so that webpages and netpage documents encountered during web browsing can be conveniently printed on the nearest netpage printer. .

The netpage system may collect, for the printer provider, fees and fees for revenue earned through publications printed on the provider's printer. Such revenue includes advertising fees, click-through fees, e-commerce commissions, and transaction fees. If the printer is owned by the user, that user becomes the printer provider.

Each user also has a netpage account 820 that is used to earn micro-debits and credits (as described in the paragraph above); Connection details 815 including name, address and telephone number; Global preferences 816 including privacy, delivery, and localization settings; A predetermined number of biometric records 817 including a user's encoded signature 818, fingerprint 819, and the like; A handwriting model 819 maintained automatically by the system; And a SET payment card account 821 in which an e-commerce payment is made.

2.3.2 Favorites List

Netpage users can maintain a list of "favorites" that link useful documents and the like on the netpage network. The list is maintained by the system for the user. The list consists of a hierarchy of folders 924, with a preferred embodiment shown in the class diagram of FIG.                 

2.3.3 History List

The system maintains a history list 929 for each user that includes links to documents and the like that users access through the netpage system. It is configured as a date-ordered list, with the preferred embodiment shown in the class diagram of FIG.

2.4 INTELLIGENT PAGE LAYOUT

The netpage publishing server automatically lays out each user's personalized publication on a section-by-section basis. Most advertisements are in the form of pre-formatted rectangles, so they appear on the page before editorial content.

The advertisement rate of the section is obtained as the rate that varies widely on each page within the section, which is used by the ad layout algorithm. The algorithm attempts to co-locate closely related editorial and advertising content, such as placing advertisements on roof repairs in its publications because of the special features of do-it-yourself roofing repairs. It consists of a configuration.

The edits selected for the user, including text and associated images and graphics, are then laid out according to various aesthetic principles.

The overall procedure, including ad selection and editorial content selection, should be repeated to bring the user closer to the section size preferences mentioned by the user once the layout has been focused. However, section size preferences can be tailored to the temporal average while allowing for significant day-to-day variations.

2.5 DOCUMENT FORMAT

Once the document is laid out, the document is encoded for efficient distribution and persistent storage on the netpage network.

The main efficiency mechanism is to separate information specific to a single user's edition from information shared across multiple users' editions. Specific information consists of page layouts. Shared information consists of objects that include images, graphics, and text that the page layout refers to.

The text object contains fully-formatted text expressed in Extensible Markup Language (XML) using Extensible Stylesheet Language (XSL). XSL provides precise control over text formatting independent of the area in which text is set (in this case provided by an array). Text objects include embedded language code that enables automatic translation and embedded hyphenation hints to help with paragraph formatting.

The image object encodes the image in JPEG 2000, a wavelet-based compressed image format. Graphic objects encode two-dimensional graphics in Scalable Vector Graphics (SVG) format.

The layout itself consists of a series of image and graphic objects, linked textflow objects through which text objects pass, hyperlinks and input areas and watermark regions as described above. These layout objects are summarized in Table 3. The layout uses a compact format suitable for efficient distribution and storage.

Table 3-Netpage Layout Objects

Layout object property The format of the linked object image location - Image object ID JPEG 2000 graphic location - Graphic object ID SVG Text flow Textflow ID - area - Optional text object ID XML / XSL Hyperlink form - area - Application ID, etc. - domain form - meaning - area - Watermark area -

2.6 DOCUMENT DISTRIBUTION

As mentioned above, for efficient distribution and persistent storage on the netpage network, the user-specific page layout is separated from the shared object it refers to.

When the subscription publication is ready to be distributed, the netpage publishing server assigns a unique ID to each page, page instance, document and document instance with the help of the netpage ID server 12.                 

The server computes a set of optimized subsets of shared content, generates a multicast for each subelement, and names the multicast channel to carry the shared content used by that array. To the layout specific to each user. The server then pointscast each user's layout to that user's printer via the appropriate page server, and multicasts the shared content on a particular channel when pointcasting is complete. After receiving the pointcast, each page server and printer subscribes to a multicast channel specific to that page layout. During multicast, each page server and printer extracts an object that the page layout refers to from a multicast stream. The page server constantly preserves page layout and shared content.

Once the printer receives all the objects that the page layout references, the printer recreates, rasterizes, and prints the fully-populated layout.

Under normal circumstances, the printer prints the page faster than it is delivered. Assuming that a quarter of each page is an image, the average page has a size of 400 KB or less. Thus, a printer can accommodate more than 100 such pages in 64MB of internal memory, considering temporary buffers and the like. The printer prints at 1 page per second. This is equivalent to 400 KB per second or approximately 3 Mbits of page data, which is similar to the expected maximum speed of page data delivered over a broadband network.

Even under abnormal circumstances, such as when the printer runs out of paper, the user will be able to replenish the paper feed before the printer's internal storage capacity of 100 pages is exhausted.

However, if the printer's internal memory is full, the printer will not be able to use it when the first multicast occurs. The netpage publishing server thus allows the printer to submit re-multicast requests. If a threshold number of requests is received or elapses, the server remulticasts the corresponding shared object.

Once the document is printed, the printer can create an exact copy at any time by retrieving page layout and content from the relevant page server.

2.7 ON-DEMAND DOCUMENTS

If a netpage document is requested by an order, it can be personalized and delivered in much the same way as in a regular netpage document. However, since there is no shared content, delivery is done directly to the requesting printer without using multicast.

If a non-netpage document is requested by an order, it is not personalized and is delivered via a netpage formatting server designated to reformat it into a netpage document. The netpage formatting server is a special instance of the netpage publishing server. The Netpage Formatting Server has knowledge of various Internet document formats, including Adobe's Portable Document Format (PDF) and Hypertext Markup Language (HTML). In the case of HTML, it can use a high-resolution printed page that displays a web page in a multi-column format with a table of contents. It can automatically include all webpages that link to the requested page directly. The user can adjust this behavior by preference.

The netpage formatting server allows standard netpage behavior, including interactivity and persistence, to be applicable to any Internet document, regardless of its origin and format. It hides knowledge of different document formats from both netpage printers and netpage page servers, and hides information of netpage systems from web servers.

3. Security

3.1 encryption (CRYPTOGRAPHY)

Encryption is used to protect sensitive information, both in storage and transmission, and to authenticate parties to some form of processing. There are two widely used ciphers: secret-key cryptography and public-key cryptography. Netpage networks use both types of encryption.

Secret-key cryptography, also called symmetric cryptography, uses the same key to encrypt and decrypt messages. The two parties wishing to exchange messages must first securely exchange their private keys.

Public key cryptography, also called asymmetric cryptography, uses two cryptographic keys. Two keys are mathematically related in that a message encrypted using one key can only be decrypted using the other key. One of these keys is public, while the other is kept secret. The public key is used to encrypt any message intended for the secret key holder. Once encrypted using the public key, the message can only be decrypted using the private key. Thus, the two parties can exchange messages securely without having to exchange private keys first. To ensure that the secret key is secure, it is common for the secret key holder to generate a key pair.

Public key cryptography is used to generate digital signatures. The secret key holder can generate a known hash of the message and encrypt the hash using the secret key. Anyone can then verify that the encrypted hash constitutes the signature of the secret key holder with respect to that particular message by decrypting the encrypted hash using the public key and verifying the hash for that message. If a signature is attached to the message, the recipient of the message can both verify that the message is true and has not changed in the course of transmission.

In order to perform public key cryptography, there must be a way to distribute the public key that prevents impersonation. This is usually done using certificates and certificate authorities. A certificate authority is a trusted third party that authenticates the relationship between a public key and someone's identity. The certification authority verifies the identity of the person by examining the identity document and then generates and signs a digital certificate containing the identity details and the secret key of the person. Anyone who trusts the certificate authority can use it with considerable certainty that the public key on the certificate is true. They only need to verify that the certificate is actually signed by a certificate authority whose public key is well known.

In most trading environments, public key cryptography is only used to generate digital signatures and securely exchange secret session keys. For all other purposes, secret key encryption is used.

In the discussion that follows, referring to the "secure" transfer of information between a netpage printer and a server, what actually happens is that the printer obtains the server's certificate, authenticates it with a certificate authority, and exchanges a secret session key with the server. To do this, you use a public key-exchange key in the certificate, and use that secret session key to encrypt the message data. It is obvious that a "session key" can have any short validity period.

3.2 NETPAGE PRINTER SECURITY

Each netpage printer is assigned at the time of manufacture a pair of unique identifiers stored in the printer's read-only memory and the netpage registration server database. The first ID 62 is published to uniquely identify the printer on the netpage network. The second ID is secret and is used when the printer is first registered on the network.                 

The first time you connect to a netpage network after the printer is installed, it generates a signature public / private key pair. It sends the secret ID and public key securely to the netpage registration server. The server compares the secret ID with that printer's secret ID stored in the database and accepts the registration if the IDs match. It then generates and signs a certificate containing the printer's public ID and public signing key, and stores the certificate in a registration database.

The netpage registration server acts as a certification authority for netpage printers because it has access to confidential information to verify printer identity.

When a user subscribes to a publication, a record is created in the netpage registration server database that authorizes that publisher to print the publication on that user's default printer or a particular printer. All documents sent to the printer via the page server are addressed to a particular user and signed by that publisher using the publisher's secret signature key. The page server verifies through the registration database that the publisher is authorized to deliver the publication to a particular user. The page server verifies the signature using the publisher's public key obtained from the publisher's certificate stored in the registration database.

As long as the request is initiated through a pen registered with the printer, the netpage registration server accepts the request to add print rights to the database.

3.3 NETPAGE PEN SECURITY                 

Each Netpage pen is assigned a unique identifier that is stored at the time of manufacture in the pen's read-only memory and the Netpage Registration Server database. Pen ID 61 uniquely identifies the pen on the netpage network.

One netpage pen can "know" a number of netpage printers, and one printer can "know" a number of pens. The pen communicates with the printer via radio frequency signals whenever it is in range of the printer. Once the pen and printer are registered, they regularly exchange session keys. Each time the pen sends digital ink to the printer, the digital ink is always encrypted using the appropriate session key. Digital ink is never sent erased.

The pen stores the session key indexed by the printer ID for every printer it recognizes, and the printer stores the session key indexed by the pen ID for every pen. Both have a large but finite storage capacity for the session key and can ignore the session key on a LRU (least-recently-used basis) if necessary.

When the pen comes into the area of the printer, the pen and the printer look for mutual recognition. If not mutually recognized, the printer determines whether the pen is to be recognized. This may be because, for example, the pen is owned by a user registered to use the printer. If the printer is supposed to recognize the pen but does not, the printer initiates an automatic pen registration procedure. If the printer is not configured to recognize the pen, it agrees with the pen to ignore the pen until it is placed in the charging cup, which is the point at which the registration process begins.

In addition to the public ID, the pen includes a secret key-exchange key. The key exchange key is also recorded at the time of manufacture in the netpage registration server database. During registration, the pen sends the pen ID to the printer, and the printer sends the pen ID to the netpage registration server. The server securely sends the session key to the printer by generating a session key for use with the printer and version. The server also sends a copy of the session key encrypted using the pen's key exchange key. The printer indexes the session key according to the pen ID and stores it internally and sends the encrypted session key to the pen. The pen indexes according to the printer ID and stores the session key internally.

Although a fake pen can impersonate a pen registration protocol, only a true pen can decrypt the session key sent from the printer.

When a previously unregistered pen is initially registered, it is of limited use until it is connected to the user. Registered but "un-owned" pens only request and fill out Netpage users and pen registration forms, register new users with new pens automatically linked, and add new pens to existing users. Licensed use only.

Because of the pen's hardware performance constraints, the pen uses secret key cryptography rather than public key cryptography.

3.4 SECURE DOCUMENTS                 

Netpage systems support the delivery of secure documents such as tickets and coupons. Netpage printers include a watermark printing function, but only upon request from a properly authorized publisher. The publisher indicates in the certificate the authority to print the watermark, which the printer can authenticate.

The "watermark" printing procedure uses an alternate dither matrix in a particular "watermark" area of the page. Contiguous back-to-back pages include a mirror-image watermark area that occupies the same space when printed. Dither matrices used in the watermark areas of odd and even pages are designed such that these areas cause an interference effect when "viewing" the printed paper.

The effect is similar to a watermark in that it is invisible when only one side of the page is seen and disappears when the page is copied by normal means.

Pages of a secure document cannot be copied using the built-in netpage copy mechanism described in section 1.9 above. This extends to copying netpages from a netpage-aware photocopier.

Security documents are usually created as part of an e-commerce transaction. Thus, they may include a picture of the user obtained when registering the personal information with the netpage registration server as described in section 2.

When a secure netpage document is provided, the recipient can verify its authority by requesting its status in the usual way. The unique ID of the secure document is valid only for the validity of the document, and the secure document ID is assigned discontinuously to prevent prediction by opportunistic counterfeiters. Secure document verification pens can be developed to embed feedback on verification failures to facilitate point-of-presentation document verification.

It is clear that neither watermarks nor user photos are secure from a cryptographic sense. They merely provide a significant obstacle to accidental forgery. Online document verification, especially online verification using a verification pen, increases the level of security where security is needed, but still does not fully defend against counterfeiting.

3.5 NON-REPUDIATION

In a netpage system, forms submitted by the user are reliably sent to the forms handler and persisted on the netpage page server. Therefore, it is impossible for the recipient to refuse receipt.

E-commerce payments made through the system, as described in section 4, also cannot be rejected by the payer.

4. ELECTRONIC COMMERCE MODEL

4.1 Secure Electronic Transactions (SET)

The Netpage system uses a Secure Electronic Transaction (SET) system as one of the payment systems. The SET developed by MasterCard and Visa is organized around a payment card, which is reflected in the term. However, much of the system is independent of the type of account used.

In the SET, the cardholder and the seller register the certificate authority and are issued a certificate including their public signing key. The certificate authority verifies the cardholder's registration details about the card issuer as appropriate and verifies the registration details of the seller for the acquirer as appropriate. The cardholder and the seller each securely store their secret signature key on their computer. During the payment process these certificates are used to mutually authenticate the seller and the cardholder and to the payment gateway.

SET has not yet been widely applied, in part because cardholders find it cumbersome to manage keys and certificates. The interim solution of managing the cardholder's keys and certificates on the server and granting the cardholder access via password has been somewhat successful.

4.2 SET PAYMENTS

In a netpage system, the netpage registration server acts as a proxy for the netpage user (ie, cardholder) in a SET payment transaction.

The Netpage system uses biometrics to authenticate users and authorize SET payment. Because the system is pen-based, the biometric information used becomes the user's online signature, which is a pen position and pressure that changes over time. Fingerprint biometrics will also be expensive, but can be used by designing a fingerprint sensor on the pen. The type of biometric information used only affects the acquisition of biometric information, but does not affect the authorization aspects of the system.

The first step in performing SET payments is to register the user's biometric information with the netpage registration server. This is done in a controlled environment, for example a bank, where the user's identity is verified and biometric information is obtained. Biometric information is obtained and stored in a registration database associated with a user's record. The user's picture may also be optionally acquired and linked to the record. After the SET cardholder registration process is completed, the corresponding private signing key and certificate are stored in the database. The user's payment card information is also stored in the netpage registration server to provide enough information to act as the user's agent in the SET payment transaction.

When the user provides biometric information, for example by signing a netpage order form to complete a payment, the printer securely sends the order information, pen ID and biometric data to the netpage registration server. The server verifies the biometric information with respect to the user specified by the pen ID, and then acts as the agent of the user in completing the SET payment transaction.

4.3 Micropayments (MICRO-PAYMENTS)

The Netpage system allows the user to conveniently pay for the printing of inexpensive documents ordered and copies of copyright documents, and if possible, the user to pay the costs incurred by printing the advertised item. It includes mechanisms for micro-payment. The latter depends on the level of subsidy already provided to the user.

When a user registers for an e-commerce transaction, a network account is created that collects micropayments. The user receives the statement on a regular basis and can pay the outstanding balance by using standard payment mechanisms.

The network account may be extended to collect subscription fees for periodicals, which may of course be provided to the user in the form of a separate bill.

4.4 TRANSACACTIONS

If a user requests a netpage in the context of a particular application, the application may embed a user-specific transaction ID 55 in the page. Subsequent input through the page is appended with the transaction ID, so that the application can obtain the proper context for that user's input.

However, when input through a page that is not user-specific occurs, the application must use that user's unique identity to establish context. A typical example is to add an item from a preprinted catalog page to the user's virtual "shopping cart". However, to protect the privacy of the user, the unique user ID 60 known to the netpage system is not exposed to the application. This is to prevent other application providers from easily associating individually accumulated behavioral data.

The netpage registration server instead maintains an anonymous relationship via a unique alias ID 65 between the user and the application, as shown in FIG. Whenever a user activates a hyperlink with the "register" attribute, the netpage page server requests the netpage registration server to convert the associated application ID 64 and pen ID 61 together to an alias ID 65. . The alias ID is then submitted to the application of the hyperlink.

The application maintains state information indexed by alias ID, and can retrieve user-specific state information without knowledge of the user's global identity. .

The system also maintains an independent certificate and private key for each of the user's applications, so that the user can use only application-specific information to sign application processing.

To assist in routing the "hyperlink" activation of the product bar code (UPC) to the system, the system records favorites applications for any number of product types for the user. .

Each application is associated with one application provider, and the system maintains an account for each application provider so that the provider's click-through fee and the like are deposited and withdrawn.

Application providers can be publishers of periodically subscribed content. The system records not only the expected frequency of publication, but also the willingness of a user to receive a subscription publication.

4.5 RESOURCE DESCRIPTIONS AND COPYRIGHT

A preferred embodiment of the inventory class diagram is shown in FIG. 40.

Each document and content object may be described by one or more inventory 842. The inventory uses a Dublin Core metadata element set designed to facilitate the discovery of electronic resources. Dublin Core metadata follows the World Wide Web Consortium (W3C) Resource Description Framework (RDF)

The inventory may identify the rights holder 920. The netpage system automatically transfers the royalty from the user to the owner when the user prints the copyrighted content.

5. Communication Protocols

The communication protocol defines an ordered exchange of messages between entities. In netpage systems, entities such as pens, printers, and servers use a suite of defined protocols to cooperatively handle the user's interaction with the netpage system.

Each protocol can be represented in the manner of a sequence diagram in which the horizontal dimension is used to represent the message flow and the vertical dimension is used to represent time. Each entity is represented by a rectangle containing the name of the entity and a vertical column representing the lifeline of that entity. While reality persists, lifelines appear as dashed lines. The lifeline appears as a double line while the entity is active. Since the protocol contemplated herein does not create or destroy an entity, the lifeline is generally truncated as soon as the entity stops participating in the protocol.

5.1 Subscription Delivery Protocol (SUBSCRIPTION DELIVERY PROTOCOL)

A preferred embodiment of a subscription delivery protocol is shown in FIG. 43.

Multiple users can subscribe to regular publications. Each user's plate shape may be laid out differently, but multiple user's plate shapes share common content, such as text objects and image objects. Thus, the subscription delivery protocol delivers the document structure to each printer via pointcast, while the shared content object delivers via multicast.

The application (ie, publisher) first obtains a document ID 51 for each document from ID server 12. Each document structure including the document ID and page specification is then sent to the page server 10 in charge of the newly assigned ID of the document. The application includes its application ID 64, subscriber's alias ID 65, and associated multicast channel name collection. The application signs the message using its private signature key.

The page server obtains the associated user ID 60, the user's selected printer ID 62 (which may be explicitly selected for that application, or may be the user's default printer) and application's certificate from the registration server Application ID and alias ID are used to do this.

The application's certificate allows the page server to verify the message signature. If the application ID and alias ID do not identify the subscription 808 together, the page server's request to the registration server fails.

The page server then assigns a document and page instance ID and forwards the page specification including the page ID 50 to the printer. It contains the relevant set of multicast channel names that the printer will listen to.

It then returns the newly allocated page ID for later reference.

Once an application distributes all document structures to a user-selected printer via an associated page server, it multicasts various subsets of shared objects to previously selected multicast channels. The page server and printer keep track of the appropriate multicast channel and receive the necessary content objects. They can then reside in the previous pointcast document structure. This allows the page server to add a complete document to its database and allows the printer to print the document.

5.2 HYPERLINK ACTIVATION PROTOCOL                 

A preferred embodiment of the hyperlink activation protocol is shown in FIG. 45.

When the user clicks on the netpage with the netpage pen, the pen communicates with the closest netpage printer 601 for that click. The click identifies the page and its location on the page. The printer already knows the ID 61 of the pen from the pen connection protocol.

The printer determines the network address of the page server 10a dealing with the specific page ID 50 via DNS. The address may be stored in the cache if the user has recently interacted with the same page. The printer then forwards the pen ID, its printer ID 62, page ID and click location to the page server.

The page server loads the page specification 5 identified by the page ID and determines (if present) the zone in which input element 58 the click is located. Assuming that the associated input element is a hyperlink element 844, the page server obtains the associated application ID 64 and the link ID 54, and retrieves the network address of the application server running the application 71 from DNS. Judge through.

The page server uses the pen ID 61 to obtain the corresponding user ID 61 from the registration server 11, assigns a globally unique hyperlink request ID 52 and hyperlinks. Construct request 934. The hyperlink request class diagram is shown in FIG. 44. The hyperlink request records the IDs of the requesting user and printer and identifies the clicked hyperlink instance 862. The page server then sends its server ID 53, hyperlink request ID and link ID to the application.

The application generates a response document according to application-specific logic and obtains a document ID 51 from the ID server 12. It then sends the document along with the page server's ID and hyperlink request ID to the page server 10b that is responsible for the document's newly allocated ID.

The second page server sends the hyperlink request ID and application ID to the first page server to obtain the corresponding user ID and printer ID 62. The first page server rejects the request if the hyperlink request has expired or is from another application.

The second page server allocates the document instance and page ID 50, returns the newly assigned page ID to the application, adds the complete document to its database, and finally sends it to the printer requesting the page specification.

The hyperlink instance may include a meaningful transaction ID 55 if the second page server includes the transaction ID in the message sent to the application. This allows the application to establish a transaction-specific context for the hyperlink activation.

If the hyperlink requests a user alias, that is, its "alias request" attribute is set, the first page server obtains not only the user ID corresponding to the pen ID, but also the application ID and the alias ID 65 corresponding to the user ID. In order to do so, both the pen ID 61 and the application ID 64 of the hyperlink are sent to the registration server 11. It includes the alias ID in the message sent to the application, allowing the application to obtain a user-specific context for hyperlink activation.

5.3 HANDWRITING RECOGNITION PROTOCOL

When the user draws a stroke on a netpage with the netpage pen, the pen communicates with the nearest netpage printer about the stroke. The stroke identifies that page and the path on the page.

The printer forwards the pen ID 61, its printer ID 62, page ID 50 and stroke path to the page server 10 in a general manner.

The page server invokes the page specification 5 identified by the page ID to determine which stroke of the input element 58, if any, has passed. Assuming that the associated input element is text area 878, the page server appends the stroke to the digital ink in the text area.

After a period of inactivity in the region of the text area, the page server sends the pen ID and pending strokes to the registration server 11 for interpretation. The registration server uses the user's accumulated handwriting model 822 to identify the user corresponding to the pen and to interpret the stroke as handwritten text. Once the stroke is converted to text, the registration server returns the text to the requesting page server. The page server appends the text to the text value of the text area.

5.4 SIGNATURE VERIFICATION PROTOCOL

Assuming that the input element with a stroke in the region of that element is the signature area 88, the page server 10 appends the stroke to the digital ink in the signature area.

After a period of inactivity in the region of the signature area, the page server sends the pen ID 61 and the processing stroke to the registration server 11 for verification. It also sends the form-related application ID 64, whose signature area is part of it, and the current data content of that form. The registration server uses the user's dynamic signature biometric 818 to identify the user corresponding to the pen and identify the stroke as the user's signature. Once the signature is verified, the registration server sends an application ID 64 and a user ID 60 to the registration server to identify the secret signing key specific to that user's application. It then uses that key to generate a digital signature of the form data and returns the digital signature to the requesting page server. The page server appends the digital signature to the signature area and sets the state of the associated form to frozen.

The digital signature includes the alias ID 65 of the corresponding user. This allows one form to obtain the signatures of multiple users.

5.5 Form SUBMISSION PROTOCOL

A preferred embodiment of the form submission protocol is shown in FIG. 46.

Form submission occurs through form hyperlink activation. It therefore follows the protocol defined in section 5.2 with some form-specific additions.

In the case of a form hyperlink, the hyperlink activation message sent by the page server 10 to the application 71 also includes the form ID 56 and the current data content of the form. If the form includes any signature area, the application will verify each by extracting the corresponding digital signature related alias ID 65 and obtaining the corresponding certificate from the registration server 11.

5.6 COMMISSION PAYMENT PROTOCOL

A preferred embodiment of the fee payment protocol is shown in FIG. 47.

In an e-commerce environment, fees and fees may be paid from an application provider to a publisher by click-through, transaction, and sale. Fees for fees and fees for fees may also be paid from the publisher to the printer provider.

The hyperlink request ID 52 transfers the fee or fee from the target application provider 70a (i.e. seller) to the source application provider 70b (i.e. publisher) and also the source. It may route from the application provider 70b to the printer provider 72.

The target application receives the hyperlink request ID from the page server 10 when the hyperlink was first activated as described in section 5.2. When the target application requires payment from the source application provider, it sends an application provider credit along with the hyperlink request ID to the original page server. The page server uses the hyperlink request ID to identify the source application and credits the associated registration server 11 with the source application ID 64, its server ID 53 and the hyperlink request ID. send. The registration server credits the account 827 of the corresponding application provider. It also notifies the application provider.

If the application provider needs to credit the printer provider, it sends the printer provider credits with the hyperlink request ID to the original page server. The page server uses the hyperlink request ID to identify the printer and sends a credit with the printer ID to the associated registration server. The registration server credits the corresponding printer provider account 814.

The source application provider is optionally notified of the identity of the target application provider, and the printer provider is notified of the identity of the source application provider.

6. Netpage Pen Description

6.1 PEN MECHANICS

8 and 9, a pen, generally indicated by reference numeral 101, is a plastic having a wall 103 that forms an interior space 104 for mounting pen components. And a housing 102 in the form of a molding. The pen top 105 is operatively mounted to one end 106 of the housing 102. A semi-transparent cover 107 is secured to the other end 108 of the housing 102. In addition, the cover 107 is a mold plastic and is formed of a translucent material so that a user can see the state of the LED mounted in the housing 102. The cover 107 includes a main part 109 substantially enclosing the other end 108 of the housing 102 and a corresponding slot formed in the wall 103 of the housing 102 protruding rearward from the main part 109. and a projecting portion 110 that fits within the corresponding slot 111. A radio antenna 112 is mounted behind the protrusion 110 in the housing 102. The screw thread 113 surrounding the aperture 113A formed in the cover 107 is formed to receive a metal piece 114 comprising a corresponding screw thread 115. The metal piece 114 may be separated to replace the ink cartridge.

In addition, a tri-color status LED 116 on the flex PCB 117 is mounted in the cover 107. The antenna 11 is also mounted on the flex PCB 117. For good all-around visibility, the status LED 116 is mounted on top of the pen 101.

The pen functions as both a typical marking ink pen and a non-marking stylus. An ink pen cartridge 118 having a nib 119 and a stylus 120 having a stylus nib 121 are mounted side by side within the housing 102. The ink cartridge nib 119 or stylus pen nib 121 may come forward through the open end 122 of the metal piece 114 by rotation of the pen top 105. Each slider block 123, 124 is mounted to the ink cartridge 118 and stylus 121, respectively. A rotatable cam barrel 125 is secured to the pen top 105 during operation and formed to rotate with the pen top. The cam body 125 includes a cam 126 in the form of a slot in the wall 181 of the cam body. Cam followers 127 and 128 protruding from the slider blocks 123 and 124 fit into the cam slots 126. Upon rotation of the cam body 125, the slider blocks 123, 124 move relative to each other to protrude either the nib 119 or the stylus nib 121 of the pen through the opening 122 of the metal piece 114. . The pen 101 has three operating states. The state produced by rotating the upper portion 105 in 90 ° steps is as follows.

Protruding the nib 121 of the stylus 120,

Protruding the nib 119 of the ink cartridge 118 and

Neither the nib 119 of the ink cartridge 118 nor the nib 121 of the stylus 120 protrudes.

The second flex PCB 129 is mounted to the electronic chassis 130 that sits in the housing 102. The second flex PCB 129 mounts an infrared LED 131 that emits infrared light to the surface for projection. An image sensor 132 is mounted to the second flex PCB to receive the reflected light reflected from the surface. In addition, the second flex PCB 129 includes a radio frequency chip 133 including an RF transmitter and an RF receiver, and a controller chip 134 for controlling the operation of the pen 101. Mount it. An optics block 135 (formed from molded clear plastics) is seated in the cover 107 to project an infrared beam to the surface and receive an image on the image sensor 132. A power supply wire 136 connects the components on the second flex PCB 129 to a battery contact 137 mounted within the cam body 125. Terminal 138 is connected to battery contact 137 and cam body 125. A 3 volt rechargeable battery 139 is seated in the cam body 125 in contact with the battery contacts. An induction charging coil 140 is mounted around the second flex PCB 129 to charge the battery 139 through induction. In addition, the second flex PCB 129 is the pressure applied to the surface by the pen nib 119 or stylus nib 121 of the pen when either the stylus 120 or the ink cartridge 118 is used for recording. An infrared LED 143 and an infrared photodiode 144 for detecting a displacement in the cam body 125 may be mounted to determine. The infrared photodiode 144 detects light from the infrared LED 143 via reflectors (not shown) mounted on the slider blocks 123 and 124.

Rubber grip pads 141, 142 are provided on the other end 108 side of the housing 102 to help hold the pen 101, and the top 105 also holds the pen 101 in the pocket. It includes a clip (142) for making.

6.2 PEN CONTROLLER                 

The pen 101 is formed to determine the position of the nib (stylus nib 121 or ink cartridge nib 119) by illuminating the surface area around the nib in the infrared spectrum. The pen records the location data from the nearest location tag and uses the optics 135 and control chip 134 to distance the location from the location tag to the nib 121 or nib 119. Will be calculated. The control chip 134 calculates the orientation of the pen and the nib-to-tag distance from the pen tip to the perspective distortion observed on the image tag.

The control data from the location tag instructs the pen to activate its "active area" LED (this is actually one mode of the tri-color LED 116, which the pen determines that the area being imaged is the "active area"). Yellow at the time). Thus, the area corresponding to a button or hyperlink on the surface may be encoded to activate this LED to provide the user of the pen with visual feedback that they are active when the pen 101 crosses the button or hyperlink. . The control data can also instruct the pen to capture the curb pen pressure reading. Thus, an area on the surface corresponding to the signature input area may be encoded to capture continuous pen pressure.

Pen motion in relation to the surface may include a series of strokes. One stroke consists of a time-stamped pen position on a series of surfaces that is initiated as a pen down event and completed as a pen up event. The pen pressure can be interpreted in relation to the threshold to indicate whether the pen is "up" or "down", as well as interpreted as a continuous value such as, for example, when the pen captures a signature. A series of captured strokes constitute what is called "digital ink." Digital inks can be used with computer systems to form the basis of digital exchange of drawings and handwriting for handwriting online recognition and signature online verification.

Using the RF chip 133 and the antenna 112, the pen 101 may transmit digital ink data (encrypted for security and packetized for efficient transmission) to a computing system.

If the pen is within range of the receiver, digital ink data is transmitted as it is formed. When the pen 101 moves out of range, the digital ink data is buffered in the pen 101 (the pen 101 circuitry is buffered to store digital ink data for about 12 minutes of pen movement on the surface). May be sent later).

The control chip 124 is mounted to the second flex PCB 129 of the pen 101. 10 is a block diagram illustrating the structure of the control chip 134 in more detail. 10 also shows a representation of an RF chip 133, an image sensor 132, a three-color state LED 116, an infrared light emitting LED 131, an infrared pressure sensor 143, and a pressure sensor photodiode 144. Include.

The pen control chip 134 includes a control processor 145. The bus 146 may exchange data between components of the control chip 134. Flash memory 147 and 512 KB DRAM are also included. An analog-to-digital converter 149 is provided to convert the analog signal from the pressure sensor photodiode 144 into a digital signal.                 

An image sensor interface 152 interfaces the image sensor 132. The transceiver controller 153 and the base band circuit 154 also include an RF chip 133 and an antenna (RF circuit 155 and an RF resonator). It is included to interface the inductor 156 connected with 112.

The control processor 145 acquires and decodes position data from the tag through the image sensor 132 from the surface, monitors the pressure sensor photodiode 144, controls the LEDs 116, 131, 143, and the wireless transceiver 153. Deals with short-range wireless communication. Control processor 145 is a medium-performance (-40 MHz) general purpose RISC processor. The processor 145, digital transceiver components (transceiver controller 153 and baseband circuit 154), image sensor interface 152, flash memory 147, and 512 KB DRAM 148 are single controller single controller ASICs. ASIC). Analog RF components (RF circuit 155, RF resonant circuit and inductor 156) are provided as separate RF chips.

The image sensor is an approximately 215 × 215 pixel CCD equipped with an infrared filter (such a sensor manufactured by Matsushita Electronic Corporation, Itakura, KT Nobusada, N. Okusenya, R. Nagayoshi, M. Ozaki, described in "A 1mm 50k-Pixel IT CCD Image Sensor for Miniature Camera System", IEEE Transactions on Electronic Devices, Vol. 47, number 1, January 2000.).                 

The controller ASIC 134 enters a quiescent state when the pen 101 is not in contact with the surface after a period of inactivity. It is implemented with a dedicated circuit 150 that monitors the pressure sensor photodiode 144 and wakes up the controller 134 via a power manager in the event of a pen-down.

The radio transceivers communicate in the 900 Mhz unlicensed band or 2.4 GHz unlicensed industrial, scientific and medical (ISM) band typically used by cordless telephones, to provide interference-free communication. Use frequency hopping and collision detection.

6.3 PEN OPTICS

As described above, pen optics are implemented as molded optics bodies 135. An optical lens implemented as the optical lens body 135 is schematically shown in FIG. 67. The optical lens includes a first lens 157, a mirror 158, a beam splitter 159, an objective lens 160, and an image sensor for focusing the emission light from the infrared LED 131. 132, a second lens 161 for focusing the image.

The optical path takes a sharp image with the image sensor 132 of that portion 193 of the surface being photographed across the field of view cone 192 within the required tilt range (described below). Is designed to send. The primary focusing element is the objective lens 160. It is also used in reverse to project the illumination from the IR illumination LED 131 onto the surface in the viewing area. Since it is impractical to place both the image sensor 132 and the IR LED 131 at the focal point of the subject, the beam splitter 159 is used to split the path and individual relay lenses 157, 161 are used for each image sensor 132 in each path. And IR LEDs 131 to focus again. This may also allow different apertures to be used for both paths.

The corners of the image sensor 132 serve as field stops for the capture area, and the capture path is the resulting object space angular field of view (ie, only 20 in the application of this embodiment). It is designed to require less than). The illumination path is designed so that the illumination fills the object space observation area as uniformly as possible with maximum power by generating the same object space viewing area as the capture path.

IR LED 131 is strobe to synchronize with frame capture. By using focused illumination, short exposures and small apertures are realized. Short exposure time prevents smeared images, enabling location tag data capture during pen movement. A small aperture realizes a region of sufficient depth over the full range of surface depths due to tilting. The capture path includes an apparent aperture stop 191 for this purpose.

Since the image sensor 132 exhibits strong response over the visible and adjacent infrared regions of the spectrum, the infrared filter 163 takes precedence in the capture path, thereby allowing the surface to be printed using transparent ink in the adjacent infrared region. Capture a clear image of tag data on a surface without interference from other graphics on the surface.

6.4 PEN PROCESSING

When the pen's stylus nib 121 or ink cartridge nib 119 is in contact with a surface, the pen determines its position and orientation relative to that surface at 100 Hz, thus providing accurate handwriting recognition (Tappert, C, CY Suen and T Wakahara). Article by "The State of the Art in On-Line Hand Writing Recognition" IEEE Transactions on Patent Analysis and Machine Intelligence, Vol 12, number 8, August 1990, its contents (Included herein as a cross-reference), which indicates the relative threshold of whether the pen is "Up" or "Down", the pressure sensor photodiode 144 is utilized. As noted, the pressure may be measured in continuous values so that the full dynamics of the signature can be verified.

The pen can determine the position and orientation of the nibs 119, 121 on the surface by imaging the surface area near the nibs 119, 121 in the infrared spectrum. The pen decodes the closest tag data and calculates the position of the nibs 119 and 121 relative to the position tag from the observed perspective distortion on the image tag and the known geometry of the pen optics 135 (described later). Although the positional resolution of the tag may be low, the modified positional resolution is very high and can easily exceed the 200 dpi resolution required for accurate handwriting recognition (see reference above).

Pen action on the surface is understood as a series of strokes. The stroke consists of a sequence of time-stamp pen positions on the surface, initiated by a Pen-Down Event and completed by a sequence of Pen-Up Events. In addition, the stroke is tagged with the area ID of the surface at the start of the stroke under the change of the area ID, that is, under normal circumstances. As mentioned above, each location tag contains data indicative of a location on the surface, and the area data also represents the area of the surface on which the tag is placed.

Fig. 68 is a diagram showing position tags and stroke processing in the pen. The pen controller 134 starts with an empty stroke (164). Then, the pen tip pressure is continuously sampled through the pressure sensor photodiode 144 (165) and the pen-down state is checked (166). If the pen is in a pen-down state, the pen controller 134 grasps an image of the surface (167), locates a tag within the identified image (168), decodes tag data from the tag (169) Inferred position and orientation of the pen relative to the surface, and adds the position data to the current stroke data (171). Upon detecting a pen-up event, i.e., detecting a pen-up state after a pen-down state, as indicated by the presence of a non-empty stroke (172), the pen controller 134 Encodes the stroke data (173) and transmits the stroke data to the computer system via the RF chip 133 and the antenna 112. Then start another void stroke (164).                 

Given a reasonably fast 8-bit multiplication (3 cycles), the processing algorithm (see later) uses approximately 80% of the processor time resources when the pen is active.

If the pen is outside the range of the computer system to send it to, the pen buffers the digital ink in its internal memory. The pen transmits the digital ink that was buffered the next time it entered the range of the computer system. When the pen's internal memory is full, the pen stops capturing digital ink and lights an Error LED when the user tries to write with the pen.

Table 4 lists the components of raw digital ink sent from the pen to the computer system. 69 is a diagram showing the structure of untreated digital ink. When the pen is working offline, the digital ink buffered in the pen is stored in the same type as the digital ink sent to the system.

Table 4 Raw digital ink components

Untreated Digital Ink Components unit Accuracy (bits) range Pen ID - 128 - Nib ID - 128 - Absolute time ms 64 - Last system time ms 64 - Zone ID - 100 - Time offset ms 32 49.7 days Tag ID - 16 - x offset 20 μm S9 ± 10mm y offset 20 μm S9 ± 10mm x rotation (pitch) Degree S7 ± 90 ° y rotation (roll) Degree S7 ± 90 ° z rotation (yaw) Degree S7 360 ° z force - 8 255

When the pen 101 is connected to a computer system, the controller 134 stores the pen ID, nib ID, current absolute time, and last absolute time obtained from the system before going offline in the form of a raw digital ink header 182. Notify the system. This allows the system to calculate the flow of the pen's clock and thus timeshift the digital ink received from the pen. The pen then synchronizes the system's accurate real time clock with that real time clock. When there is more than one pen working with the computer system, the pen ID allows the computer system to identify the pen. Pen IDs are important in a system that uses a pen to identify the owner of the pen and, for example, interact with that owner in a particular direct way. In other embodiments this may not be required. Thanks to the nib ID, the computer system can identify which nib, stylus nib 121 or ink cartridge nib 119 is currently being used. The computer system can vary its behavior depending on which nib is being used. For example, if the ink cartridge nib 119 is used, the computer system may postpone producing the feedback output because instant feedback may be provided by ink markings made on the surface. Wherever the stylus nib 121 is being used, the computer system can generate instant feedback output.

At the beginning of the stroke, the pen controller 134 records, in the raw stroke header 183, the time that has elapsed since the last absolute time notified to the system. For each pen position in the stroke, the controller 134 is in the form of an unprocessed pen position 177 with x and y offsets 119 and 121 of the pen tip of the pen from the current tag and x, y, z of the pen. Record the rotation and pen tip pressure. The controller only writes in the form of a tag change 178 when the tag (identifying a tag in the area) changes. Since the tag frequency is considerably smaller than the normal position sampling frequency, the tag ID may be constant for many consecutive pen positions, or for the entire stroke if its stroke is short.

Because the pen samples its position and orientation at 100 Hz, the pen position in one stroke is inherently clocked at 100 Hz, and no explicit time stamp is required. If the pen, for example, fails to decode the tag and fails to calculate the pen's position, it must record the pen position to preserve intrinsic clocking. Thus, the pen records the location as unknown, in the form of an unknown pen location (179), so that the computer system can later interpolate the location of the pen from adjacent samples if necessary. .

Since the 32-bit time offset of one stroke has a finite range (ie, 49.7 days), the pen selectively records absolute time in the form of a time change 176 for one stroke. This is the absolute time for the time at which the next stroke time offset is measured.

Because the area ID is constant for many consecutive strokes, the pen only records the area ID as it changes, in the form of area change 180. This is the area ID implicitly associated with a later pen position.

Because the user can change the nibs 119 and 121 between one stroke and the next, the pen optionally records the nib ID in the form of a nib change 175 for one stroke. This is the nib ID that is inherently associated with subsequent strokes.

As listed in Table 5, each component of one stroke is an entropy-coded prefix.

Table 5 Unprocessed Stroke Element Prefixes

Raw stroke components Prefix Unprocessed pen position 0 Unknown pen position 10 Change tags 1100 Stroke end 1101 Zone change 11100 Nib change 11101 Change time 11110

The 10mm stroke, which lasts for one second, is two or three tags long and contains about 100 position samples, making it approximately 5500 bits in size. On-line continuous digital ink capture thus requires a maximum transfer rate of 5.5 KBps, and off-line continuous digital ink capture requires about 40 KBytes / minute of buffer memory. The pen's 512KB DRAM 48 can thus afford 12 minutes or more of continuous digital ink. Changes in time, area, and nib occur rarely, and have a negligible impact on the required transfer rate and buffer memory. The additional compression of the pen position can further reduce the transfer rate and buffer memory requirements.

Each raw stroke is encrypted using a Triple-DES algorithm (Schneier, B, Applied Cryptography, Second Edition, Wiley 1996, the disclosure is incorporated herein by cross-reference herein) and then sent to the computing system. The pen and computing system exchange session keys for this purpose on a conventional basis. In short, at 50 cycles per encrypted bit, only 0.7% of the processor's 45 times are spent to encrypt 5,500 bit strokes per second.

6.5 Other Pen Examples

In another embodiment, the pen may incorporate an infrared data association (IrDA) interface for near field communication with a base station or netpage printer.

In another embodiment, the pen 101 may mount a pair of orthogonal accelerometers on the normal plane of the pen axis. Accelerometer 190 is represented by dashed lines in FIGS. 9 and 10.

By providing an accelerometer, the pen according to this embodiment detects motion without referring to the surface location tag so that the location tag can be sampled at a lower frequency. Each location tag ID can then identify the object of interest rather than the location on the surface. For example, if the object is a user interface input element (eg a command button), the tag ID of each location tag in the area of the input element can directly identify that input element.

The acceleration measured in each direction of x and y by the accelerometer may be integrated over time to yield instantaneous speed and position.

Since the starting position of the stroke is unknown, only relative positions in the stroke can be calculated. Although the integration of the position increases the error in the sensed acceleration, the accelerometer usually has a high resolution, and the duration of the stroke where the error is increased is also short.

7. Netpage Printer Description

7.1 PRINTER MECHANICS

A vertically mounted netpage wallprinter 601 is shown fully assembled in FIG. The printer prints netpages on letter / A4 compliant media using a dual 8½-inch Memjet ™ print engine 602,603 as shown in FIGS. 12 and 12A. It uses a straight path through the dual print engines 602, 603 where the paper 604 prints both sides of the paper simultaneously in primary and full bleed.

An integral binding assembly 605 applies a strip of glue along one edge of each printed sheet so that it can adhere to the previous page when pressure is applied. This produces a final bound document 618 that can range in thickness from one to several hundred sheets.

The replaceable ink cartridge 627, coupled with the dual print engine, shown in FIG. 13 is fixed, adhesive, cyan, magenta, yellow, black and infrared. It has a bladder or chamber for storing ink. The cartridge also includes a micro air filter in the base molding. The micro air filter interfaces with an air pump 638 in the printer via a hose 639. This supplies filtered air to the print head in order to prevent infiltration of micro particles that can block the printhead nozzle into the Memjet (trademark) print head 350. By including an air filter in the cartridge, the operational life of the filter is effectively linked to the life of the cartridge. Ink cartridges are fully recyclable products that have the capacity to print and glue 3000 pages (1500 sheets).

Referring to FIG. 12, motorized media pick-up roller assembly 626 is a dual Memjet printhead from a media tray via a paper sensor on a first print engine. Push the top sheet directly into the assembly. Two Memjet (trademark) print engines 602, 603 are mounted so as to be apposing in-line sequential configuration along a straight paper path. The paper 604 is rolled into the first print engine 602 by an integrated powered pick-up roller 626. The position and size of the paper 604 is sensed and full bleed printing is initiated. Fixatives are printed simultaneously so that they can be dried in the shortest possible time.

The paper is ejected from the first Memjet ™ print engine 602 via a set of powered exit spike wheels (aligned along a straight paper path) that operate opposite the rubber rollers. These spike wheels are in contact with the 'wet' printed surface and continue to feed the paper 604 to the second Memjet® print engine 603.

12 and 12A, paper 604 passes from dual print engines 602, 603 to bookbinding assembly 605. The printed page includes a powered spike wheel axle 670 with fibrous support rollers, and another movable axle with spike wheels and momentary action glue wheels. Pass through). Movable axle / glue assembly 673 is mounted on a metal support bracket and is powered axle via a gear by the operation of a camshaft. It is moved forward to interface 670. Each motor powers this camshaft.

The adhesive wheel assembly 673 consists of a partially hollow axle 679 having a rotating coupling for a glue supply hose 641 from the ink cartridge 627. This shaft 679 is connected to the adhesive wheel, which absorbs the adhesive through the radial hole by capillary action. Molded housing 682 has an opening in the front and surrounds the adhesive wheel. Pivoting side molding and spring outer doors are attached to the metal brackets to rotate the sides out as the rest of the assembly 673 is pushed forward. This operation exposes the adhesive wheel through the front of the molded housing 682. Tension springs close the assembly and effectively cover the adhesive wheels during periods of inactivity.

As the paper 604 passes through the adhesive wheel assembly 673, the adhesive is transported down toward the booklet assembly 605 and painted on one vertical edge of the front side of the paper (except the first sheet of the document).

7.2 PRINTER CONTROLLER ARCHITECTURE

The netpage printer control unit includes a control processor 750, a factory-installed or field-installed network interface module 625, and a radio transceiver (transceiver controller 753, as shown in FIG. 14). ), Baseband circuitry 754, RF circuitry 755, and RF tuning circuitry and inductor 756), dual raster image processor (RIP) DSP 757, dual print engine controller 760a, 760b), flash memory 658, and 64MB DRAM 657.

 The control processor handles communication with the network 19 and local wireless netpage pen 101, senses a help button 617, and provides user interface LEDs 613-616. Control and feed and synchronize the RIP DSP 757 and print engine controller 760. The control processor consists of a medium performance general purpose microprocessor. The control processor 750 communicates with the print engine controller 760 via a high speed serial bus 659.

The RIP DSP rasterizes the page specification and compresses it into the compressed page format of the netpage printer. Each print engine controller dithers the page image in real time (ie, more than 30 pages per minute) and prints to the associated Memjet® printhead 350. The dual print engine controller prints both sides of the paper at the same time.

The master print engine controller 760a controls the media feed and uses ink in collaboration with the master QA chip 665 and the ink cartridge QA chip 761. Supervise.

The flash memory 658 of the print controller accommodates configuration data as well as software for both the processor 750 and the DSP 757. This is copied to main memory 657 at boot time.

Processor 750, DSP 757, and digital transceiver components (transceiver controller 753 and baseband circuit 754) are integrated into a single controller ASIC 656. Analog RF components (RF circuit 755 and RF tuning circuit and inductor 756) are provided as separate RF chips 762. The netpage printer allows both factory-selected and field-selected network connections, so the network interface module 625 is separate. Flash memory 658 and 2 x 256 MBit (64 MB) DRAM 657 are also separate off-chips. The print engine controller 760 is provided in a separate ASIC.

Various network interface modules 625 are provided such that each module provides a netpage network interface 751 and, optionally, a local computer or network interface 752. Netpage network Internet interfaces include POTS modems, Hybrid Fiber-Coax cable modems, DSL modems, satellite transceivers, current and next-generation cellular telephone transceivers. ) And a wireless local loop transceiver. Local interfaces include IEEE 1284 (parallel ports), 10Base-T and 100Base-T Ethernet (Ethernet), USB and USB 2.0, IEEE1394 (Firewire), and various emerging home networking interfaces. If Internet access is available on the local network, that local network interface can be used as the netpage network interface.

The radio transceiver 753 communicates in the unlicensed 900 MHz band or unlicensed 2.4 GHz industrial, scientific and medical (ISM) band typically used for cordless telephones, and provides frequency hopping and communication to provide interference free communication. Use collision detection.

The printer control unit optionally implements an IrDA interface to receive data from devices such as netpage cameras. In another embodiment, the printer uses an IrDA interface for near field communication with a properly configured netpage pen.

7.2.1 RASTERIZATION AND PRINTING

Once the main processor 750 receives and verifies the page array and page object of the document, it executes the appropriate RIP software on the DSP 757.

The DSP 757 rasterizes each page specification and compresses the rasterized page image. The main processor stores each compressed page image in memory. The simplest way to load-balance multiple DSPs is to have each DSP rasterize separate pages. In general, any number of rasterized pages can be stored in memory, so the DSP can remain busy at all times. This policy leads to potentially poor DSP utilization only when rasterizing short documents.

The watermark region in the netpage specification is rasterized into a contone-resolution bi-level bitmap that is losslessly compressed to form part of the compressed page with negligible size.

The infrared (IR) layer of the printed page contains encoded netpage tags at about six densities per inch. Each tag encodes a page ID, a tag ID, and a control bit, and the data content of each tag is generated during rasterization and stored in a compressed page image.

The main processor 750 passes back-to-back page images to the dual print engine controller 760. Each print engine controller 760 stores the compressed page image in its local memory and initiates a page expansion and printing pipeline. Storing 114MB two-level CMYK + IR page images in memory is impractical, so page enlargement and printing are pipelined.

7.2.2 Print Engine Controller                 

The page magnification and print pipeline of the print engine controller 760 includes a high speed IEEE 1394 serial interface 659, a standard JPEG decoder 763, a standard Group 4 fax decoder 764, a custom haftoner / compositor unit 765, custom tag encoder 766, line loader / formatter 767, and custom interface 768 to Memjet® printhead 350.

Print engine controller 360 operates in a double buffered manner. While one page is loaded into DRAM 769 via high speed serial interface 659, a previously loaded page is read from DRAM 769 and passed to the print engine controller pipeline. Once the page has finished printing, the page just loaded is printed while another page is loaded.

In the first stage of the pipeline, all at the same time, the JPEG-compressed continuous tone CMYK layer is enlarged (763) and the Group 4 Fax-compressed two-level black layer ( expand the bi-level black layer (764) and render the bi-level netpage tag layer (766) according to the tag format defined in section 1.2. In a second step, dither the continuous tone CMYK layer (765) and composite (765) the two-level black layer to the resulting two-level CMYK layer. The resulting bi-level CMYK + IR dot data is buffered and formatted for printing on a Memjet® printhead 350 via a set of line buffers. It is formatted (767). Most of these line buffers are stored in DRAM, the chip of Valdo. In the final step, six channels of two-level dot data (including fixative) are printed to the Memjet (trademark) printhead 350 via the printhead interface 768.

If several print engine controllers 760 are used together in a duplexed configuration, they are synchronized via a shared line sync signal 770. Only one print engine 760 selected via an external master / slave pin 771 generates a line sync signal 770 on the shared line.

The print engine controller 760 synchronizes the page expansion and rendering pipeline, configures the print head 350 via the low speed serial bus 773, and includes a stepper motor ( Slow processor 772 to control 675,676.

In an 8½ "version netpage printer, each of the two print engines prints 30 portrait (11") letter pages per minute at a line speed of 8.8 kHz at a resolution of 1600 dpi. In the 12-inch version Netpage printer, the two print engines print 45 longitudinal (8½ ") letter pages per minute at a line rate of 10.2 kHz each. This line rate is 30 kHz in the current design. It is sufficiently included in the operating frequency of the exceeding Memjet print head.

8. NETPAGE TAGS                 

8.1 TAG TILING

8.1.1 Planar Surface Tag Tiling

In order to support "single click" interaction with the tag region via the sensing device, the sensing device can observe at least one entire tag 4 irrespective of its position or placement in the viewing area. It should be possible. The required diameter of the viewing area of the sensing device is thus a function of the size and spacing of the tags 4.

If the shape of the tag is circular, e.g. in the case of the example tag 4 as described above, the minimum diameter m of the observation sensing area is such that the tag 500 of diameter k is tiled on the equilateral triangle grating as shown in FIG. Is defined in EQ1. This is accomplished when the center-to-center tag spacing is equal to the tag diameter k.

When the tag diameter is k and 256 dots (4 mm at 1600 dpi), m is thus 552 dots (ie 8 mm). When the monotonous area of 16 dots, ie the effective tag diameter is k and 272 dots (ie 4.3 mm), m increases to 587 dots (ie 9.3 mm).

If the tag 4 is moved away by the distance s (s is at least as large as k), the minimum viewing area is obtained as EQ2.

When no horizontal overlap is expected between successive lines of the tag 500, for example to facilitate tag rendering, the tags must be moved away from the minimum amount given by EQ3. Thus, for a 256-dot diameter tag, u is 40 dots (0.6 mm at 1600 dpi). This is beyond the monotonous area required for the tag, so monotonous areas can be ignored if the tag lines are rendered so that they do not overlap.

In EQ2, setting s = k + u results in EQ4. Thus, for a tag of 256 dots diameter, s would be 296 dots (4.7 mm at 1600 dpi) and m would be 598 dots (9.5 mm).

8.1.2 Spherical Surface Tag Tiling

Common icosahedrons are often used as the basis for creating triangular tiling close to a sphere. A typical icosahedron, such as the icosahedron 526 of FIG. 53, shares twenty corners 530 and twelve vertices 532 and twenty equilateral sides such that five corners 530 meet at each vertex 532. It is made up of triangles.

In order to achieve the desired tiling, the icosahedron 526 is engraved on the target sphere, and each triangle 528 of the icosahedron 526 is subdivided into equal numbers of equally sized equilateral triangles to achieve the desired total number. Create a triangle of. If each corner 530 of the icosahedron is divided into ν equidistant intervals and defines a set of ν-1 points along each corner, each pair of corresponding points along two neighboring corners are shared differently. When connected by parallel to the edge of the neighboring line, so drawn line is ν ² of the triangle, i.e. all 20ν each triangular face 528 of the icosahedron 526, as the horizontal diameter of the granularity for the vertex of the equilateral triangle of the desired same size the ^two triangles is obtained. Of the resulting 10v ² +2 vertices, five triangular faces meet at each of the twelve original vertices of the icosahedron 526, and six triangular faces meet at each of the remaining vertices. The twelve original vertices 532 are on the sphere, while the remaining vertices are inside the sphere. Each generated vertex then protrudes centered on the sphere while providing the desired tiling.

In this way, a sphere approximated as a general polyhedron is called geodesic, and the parameter v is called the frequency of the geodesic. Fig. 54 shows the icosahedral geodesic line 534, i.e., the 180 icosahedron 528, with ν 3.

The closer the subdivision triangle is to the center of the face of the icosahedron 526, the farther away from the surface of the sphere, the larger it is when projecting on the sphere. To minimize the variation in the size of the projected subdivision triangles, the detail vertices are structurally removed prior to extrusion (Tegmark, M., "An Icosahedron-Based Method for Pixelizing the Celestial Sphere", ApJ Letters, 470, L81, October 14, 1996). If ν = 1, no vertex is created and the angle to the triangle face remains 60 ° from the vertex. As ν increases, however, the surface defined by the five triangular faces and surrounding each original vertex becomes gradually flat, and the vertex angle of each triangular face is 72 ° (ie 360 ° / Converge to 5). This is the worst case of tag tiling of a sphere. In the isosceles triangle at 72 °, the base length is 1.18 times the length of the other two sides. The maximum value s of the tag spacing for the purpose of calculating the observation detection area is thus close to 1.18k. With a tag diameter of 256 dots and a monotonous region of 16 dots, that is, the effective tag diameter k is 272 dots (4.3 mm), m becomes 643 dots (10.2 mm) according to EQ2.

The angle to each corner of the icosahedron at the center of the circumscribed circle is given by EQ5.

For a sphere of radius r, the length of the arc of the corner where each central portion protrudes is rθ. When the tag diameter, which is the same unit as r, is given by K, the number n of tags for wrapping the sphere is given by EQ6.

For a given n, r is limited by EQ7.

If n is constrained to 2 ¹⁶ , and you do not need multiple areas to enclose the sphere, and you want to use 16-bit tag IDs with K as 4.3 mm above, then r is limited to 310 mm.

Typical spheres have a radius of 160 mm. Its protruding arc length (177 mm) fits into 41 equally spaced tags with negligible additional spacing. Such a sphere uses a total of 16812 tags.

8.1.3 Arbitrary Curved Surface Tag Tiling

Triangular meshes provide a local scale of the surface with local curvature and error ranges, without discontinuity or specificity for surfaces with arbitrary topography and topology. error bound) to approximate it. Assuming a triangular mesh for a particular surface exists, effective non-regular tag tiling can be made as long as each mesh triangle relates to a minimum vertex angle and minimum corner length. If the viewing area of the sensing device is guaranteed to contain at least one complete tag at any position of the sensing device on the surface, the tiling is treated as effective in relation to the particular sensing device.

A tiling precedure is initiated by placing a tag at each vertex of the mesh so that the minimum edge length is equal to the tag diameter k. The tiling procedure continues by inserting the tag in the center of any corner of length exceeding the maximum tag separation (s). As shown in Fig. 9, the maximum tag spacing s is calculated so that if two neighboring tags 4a and 4b are separated by a distance s + ε there is a space for another tag 4c between them. (Ie EQ8).

However, if the vertex angle between two edges of length (s + ε) is less than 60 °, the inserted tags will overlap.

In order to prevent duplication of the inserted tags, the minimum tag spacing t is introduced when t≥k. The minimum vertex angle α is then a function of k and t as shown in EQ9.

Clearly, if t = k, β is constrained to 60 degrees. That is, the mesh is constrained equilateral. However, as shown in Fig. 56, if t > k,? Can be made smaller than 60 degrees without overlapping the inserted tag.                 

The maximum tag spacing s should be based on the new minimum tag spacing t in EQ10.

Given a particular mesh triangle, there are four different scenarios for inserting tags. Assuming that the minimum vertex angle is not less than 30 ° (i.e. half of 60 °), each time the mesh triangle has at least one corner whose length is less than or equal to s, the remaining two corners are less than 2s long. In fact, the minimum vertex angle is usually at least 45 °.

In the first scenario (FIG. 57), no edge of triangle 546 is greater than s in length, so tagging of the triangle is already complete.

In the second scenario (FIG. 58), the length of one corner 548 of triangle 550 exceeds s. Tag 552 is inserted in the center of edge 548 to complete tagging of triangle 550.

In the third scenario (FIG. 59), the length of the two edges 554, 556 of the triangle 558 exceeds s. The tags 560 and 562 are inserted in the middle of the two long edges 564 and 566 to complete the tagging of the triangle 558. The center of the two inserted tags 560, 562 forms a trapezoid with the two vertices 564, 566 of the short edge 568 of the original triangle 558. If any of the trapezoidal diagonals exceeds s, the final tag 570 is inserted at the center of the diagonals to complete the tagging of the triangles.

In a fourth scenario (FIG. 60), the length of all three corners 572 of triangle 573 exceeds s. Tagged vertices 574 are inserted in the center of each corner 572 and three new vertices 574 are connected by edge 576. The tagging stop is then iteratively applied to the four resulting triangles 577,578,579,580 respectively. Note that the new triangle has the same shape as the original triangle 573, so that it maintains the minimum vertex angle.

The parameters of tag tiling are arranged as shown in FIG.

[Figure 4] Tag Tiling Variables

variable meaning β Minimum vertex angle k Tag diameter m Minimum diameter of sensor field of view on surface s Maximum center-to-center tag spacing t Minimum center-to-center tag spacing

8.2 TAG SENSING

8.2.1 Pen Orientation

In order for a pen-like sensing device to be used as a convenient writing instrument, a pen-direction ranger must be supported. Since the nib is in contact with the surface, the orientation of the pen is characterized as yaw (z rotation), pitch (x rotation) and roll (y rotation) of the pen as shown in FIG. Can lose. Since yawing of the pen cannot be generated, it is appropriate to generate the pitching and rolling of the pen along with the overall tilt of the pen due to the pitching and rolling.

Yawing is usually applied after pitching, for example in the case of a pen input tool, yawing allows to define a twist around a physical axis rather than a direction in the surface plane. In pens with a writing nib, however, the image sensor is mounted off the axis of the pen, so that the pen's image sensing capability (and thus yawing detection, as described below) is as long as the pen is almost vertical. Ability) is constrained. Yawing is thus applied before pitching, allowing a full yaw range of yawing specified by the rotation of the pen relative to the surface while keeping the peering and rolling constant.

Peering and rolling are usually defined as y and x rotations, respectively. Here they are defined as x and y rotations, respectively, because they are defined in terms of the x-y Cartesian coordinate system of the surface, and when viewed by the user, the y axis naturally becomes the vertical axis and the x axis naturally becomes the horizontal axis. In right-handed 3D coordinate systems, rolling is usually defined as positive when anticlockwise, while pitching and yawing are usually defined as positive when clockwise. Herein, all rotation directions are defined as being positive when they are counterclockwise.

The overall tilt of the pen is related to its pitching and rolling and as in EQ11.

The tilt of the pen affects the scale at which the surface shape is imaged at different points in the viewing area, and thus the resolution of the image sensor. Detecting the area directly below the nib is impractical, so the tilt of the pen affects the distance from the nib to the center of the image area. This distance must be known to ensure that the correct nib position is derived from the position determined from the tag.                 

8.2.2 Image Sensing

The viewing area is the cone defined by the solid half-angle (α) (when each angular field of view is 2α) and D on the surface when the optical axis is perpendicular. It can be modeled as the top height of. The image sensor is usually rectangular, but it has been previously noted that only the maximum elliptical subarea of the image sensor is involved in ensuring that a sufficiently large portion is taken.

The intersection of the surface and the field of view cones defines an elliptical window on the surface. This window is circular when the optical axis is vertical.

FIG. 62 shows the geometric relationship between the pen tip (point A) for a given pitch-related tilt θ, the optical axis CE of the pen, and the viewing area window FH. The slope is defined as positive clockwise from the vertical line. The following equation applies to both positive and negative slopes.

If the pen is not tilted, the window diameter (ie | BD |) is given by EQ12.

If the pen is not tilted, the distance from the nib to the edge of the window (ie | AB |) is T, so the distance S from the nib to the center of the window (ie | AC |) is given by EQ13.

When the pen is tilted by θ, the distance from the viewpoint to the surface along the optical axis decreases to d (ie | GE |) and is given by EQ14.                 

The width of the window (ie | FH |) is given by EQ15.

D and α should be chosen so that an adequately wide area can be photographed through the supported tilt range. The required minimum diameter (m) of the area is given by EQ4, while the actual shooting area width is given by EQ15. This gives you EQ16.

Once D and α are determined, the image sensor resolution should be selected so that the imaging area is sampled appropriately, that is, the maximum feature frequency is sampled at or above the Nyquist rate.

When photographing, the scale of the surface decreases as the distance from the viewpoint increases and also as the slope to the observation beam increases. These factors all have the greatest effect when at point F at the positive slope and at point H at the negative slope, that is, when the point in the window is farthest from the viewpoint. It should be noted that in the discussion below, reference to F applies to H when the slope is negative.

The distance from the observation point to point F (ie | EF |) is given by EQ17.

The scaling according to the slope of the surface with respect to the observation light (EF) passing through F is given by EQ18.

If the surface characteristic frequency is f, then the angular surface feature frequency (ω) (i.e. in relation to the viewing area) at F according to the two factors is given by EQ19.

If there is no slope in the object plane (ie θ = 0), this is simplified to EQ20.                 

The image sensor, by definition, is required to photograph at least the entire angular field of view. Since the pixel density of the image sensor is constant, the entire observation area should be taken at the maximum frequency. Given each observation area in the 2α 'image area, the image sensor tilt (ie image plane tilt) with respect to the optical axis θ' and the sampling rate by n (n≥2 according to the Nyquist theorem) and the minimum image sensor The resolution q is given by EQ21 and EQ22.

The square cosine of the molecule in EQ22 comes from the same reason as the square cosine of the denominator in EQ19.

If there is no slope of the image plane (i.e., θ '= 0), and the angular fields of view of the image space and object space are the same (i.e., α' = α), This is simplified to EQ23 and EQ24.

If there is no slope of the target plane (ie θ = 0), this is further simplified to EQ25.

EQ22 is simplified to EQ26 if the image plane slope and the destination plane slope are equal (i.e., θ '= θ), and each observation inverse of image space and object space is the same (i.e., α' = α).

Therefore, matching the image plane tilt to the destination plane tilt can achieve a smaller required image sensor size than when the image sensor tilt is fixed at zero and eliminates perspective distortion from the captured image. . However, variable image sensor tilt is a relatively expensive option to put to practical use and also requires a deeper area.                 

FIG. 63 shows the geometric relationship between the pen tip (point A) for a given rolling related tilt θ of the optical axis of the pen, the optical axis CE of the pen, and the viewing area window FH. The slope is likewise defined as being positive clockwise from the vertical line. As an exception to EQ13, the above equation applies equally to roll-induced tilt. The distance S from the pen tip to the center of the window (ie | AC |) for the tilt due to rolling is zero, as defined by EQ13.

For tilt due to pitching, the magnitude of the tilt range can be maximized by selecting the minimum (negative) and maximum (positive) slopes with the same image sensor requirements. For tilt due to pitching, the minimum has a smaller value than the maximum because the surface is farther away from the positive slope of the same magnitude than the distance from the negative slope. The same is true for the slope due to rolling.

As described above, the smallest characteristic of the tag 4 also has a structure for encoding data bits, and has a minimum diameter of 8 dots. This gives a maximum characteristic frequency f of about 7.9 / mm at 1600 dpi.

As previously calculated according to EQ4, 256-dot diameter equilateral triangular tiling without overlap between successive lines of tags requires a minimum viewing area window on the surface of 598 dots, that is, a viewing area window of about 9.5 mm at 1600 dpi.

Most people hold the pen with a pitch of about + 30 ° and a rolling of 0 °. The ink ball of the ballpoint pen nib loses effective contact with the surface beyond a pitch of about + 50 ° (40 ° from the horizontal). The proper target pitching range is thus from -10 ° to + 50 °, and given the greater constraints of the combination of pitching and rolling given by EQ11, the suitable rolling range is from -30 ° to + 30 °.

Highly compact (1.5mm ² ) Matsushita CCD image sensor (Matsushita Electric Corporation, Itakura, KT Nobusada, N Okusenya, R Nagayoshi, M Ozaki, "A 1mm 50k-Pixel IT CCD Image Sensor for Miniature Camera System", IEEE Transactions on Electronic Devices, Volt 47, number 1, January 2000) is suitable for use in dense devices such as pens. It has a possible resolution of approximately 215 x 215 pixels. Image and Objective Assuming that each viewing area is the same, there is no image plane tilt, and the nib-window distance T is 4 mm, geometry is achieved using EQ16 and EQ24 to achieve the required pitching and rolling range as described above. By optimizing the pitching range from -16 ° to + 48 ° (64 °) and the rolling range from -28 ° to + 28 ° (56 °), the viewing distance 9D is 30nm and each viewing area is 18.8 ° (α = 9.4 °). The possible pitching range is practically from -21 ° to + 43 °, which can be mapped to the desired range by pitching the optical axis -5 ° in relation to the physical axis. It should be noted that the tilt range can be somewhat expanded by optimizing the non-zero tilt of the image plane.

The overall pen tilt is therefore constrained to an elliptical cone with a major angle of 64 ° in the pitch plane and a 56 ° minor angle in the rolling plane.

Image sensing parameters are summarized in Table 5.                 

[Table 5] Image Sensing Variables

variable meaning α Object space field of view half-angle α ' Image space field of view half-angle γ Pen yaw θ Tilt objective plane (i.e. global pen tilt) θ ' Tilt image plane φ Pen pitching ψ Pen rolling ω Each frequency in the observation plane D Normal viewing distance d Tilted viewing distance f Surface characteristic frequency n Sampling rate q Image sensor resolution S Distance from nib to center of surface viewing area (when θ = 0) T Distance from nib to edge of surface viewing area (when θ = 0)

8.3 TAG DECODING

8.3.1 Tag Image Processing and Decoding

Tag image processing is described above in Section 1.2.4. It is at its peak by understanding the decoded tag data as well as the 2D perspective transform on the tag.

8.3.2 Inferring Pen Transform

Once a 2D perspective transformation is obtained that represents a perspective transformation of a tag in the captured image, a corresponding discrete 3D tag transform with respect to the optical axis of the pen can be inferred as described above, see section 8.4 below. This is explained in

Once a discrete 3D tag transformation is recognized, the corresponding 3D pen transformation, ie the transformation of the pen's physical axis relative to the surface, can be inferred. The physical axis of the pen is experienced by the pen user as an axis inherent in the shape of the pen. This passes through the nib. The relationship between the physical axis and the optical axis is shown in FIG.

It is convenient to define three coordinate spaces. In sensor space, the optical axis coincides with the Z axis and the viewpoint is the origin. In pen space, the physical axis coincides with the z axis and the pen tip is the origin. In tag space, the tag 4 exists on its x-y plane with its center as the origin. Tag conversion converts the tag 4 from tag space to sensor space.

Sensor space is shown in FIG. 64. The sign of the dot of FIG. 64 is the same as that of FIG. The observation point is E, the detected point is G, and the pen tip is A. The point of intersection between the optical axis and the surface is G and is called a sensed point. Compared to the geometry shown in FIG. 62 where the nib is treated as a point, the nib is treated here as a small sphere. If the nib is bent, the tilt of the physical axis affects the offset between the detected point and the contact point between the nib-surface. The central point K of the spherical nib approximately around the physical axis pivot is referred to as the pivot point.

The nib makes a nominal contact with the surface at point A when the optical axis is perpendicular. KA is defined as being parallel to the optical axis. However, when the pen is tilted, contact is made at point L as shown in FIG. Given the radius R of the pen tip, the distance from the surface of the pivot point K, ie A or L, is always R.

Discrete tag transformations include translation of the tag center from the sensed point, 3D tag rotation, and movement of the sensed point from the viewpoint.                 

Given the shift d from the viewpoint to the sensed point in discrete tag transformation, according to EQ14, the sensed point is given as EQ27.

Since the physical axis differs only in y movement and x rotation (ie pitching) from the optical axis, the physical axis lies on the y-z plane. Referring to FIG. 64, when | AC | = S and | EC | = D (ie, as in FIG. 62), it is evident that the position of the pivot point in sensor space is given by EQ28.

The vector from the detected point to the pivot point is therefore given by EQ29.

The bet from the pivot point to the contact point is by definition a surface normal of length R. It is done as shown in EQ30 and EQ31 by applying the 3D tag rotation (M) to the tag space surface normals, normalizing the results and scaling with R.

The vector from the detected point to the contact point is then obtained with EQ32.

It is transformed into tag space by applying the inverse of the tag transformation 3D rotation, which is then added to the vector from the tag center to the sensed point according to EQ33, so that the tag space, i.e. from the surface to the contact point of the tag Provide a vector.

This is finally added to the absolute location of the tag, providing the desired absolute location of the pen tip in the tagged area, as implied by its tag ID (see EQ34).

The final step is to infer the 3D direction of the pen from the 3D direction of the tag. As defined in EQ35, EQ36, and EQ37, the discrete rotation of the pen is simply the inverse of the discrete rotation of the tag, and the pitching of the pen also affects the pitch of the optical axis (φ _sensor ) relative to the axis of the pen. It includes.

8.4 INFERRING THE TAG TRANSFORM

The image of the tag 4 captured by the image sensor includes perspective distortion due to the position and orientation of the image sensor relative to the tag. Once the perspective target of a tag is found in image space, a perspective transform with eighth degree of freedom is inferred by solving an easy-to-understand equation for four tag-space and image-space point pairs. The discrete transform steps that produce the image of the tag are symbolically associated, and a set of concurrent discontinuity equations match the associated transform and the corresponding elements in the perspective transform. Solving these equations provides a discrete transformation step, which includes a nib, 3D tag rotation, and a desired tag offset from the viewpoint offset from the surface.

8.4.1 Modeling the Tag Transform

The transformation of tag 4 from tag space to image space can be modeled as a continuation of the next transformation step.

X-y movement (by tag-point offset)                 

Z rotation (by tag yawing)

X rotation (by tag pitching)

Y rotation (by tag rolling)

Z movement (by tag-point offset)

Perspective project (with a certain focal length)

X-y scale (in observation size)

These are coded and associated to create a single transformation matrix that affects tag transformations. Table 7 summarizes the discrete conversion variables used in the sections that follow, along with the scope of each variable.

[Table 7] Discrete transformation variables and their ranges

variable Abbreviation meaning Unit Conversion range γ - Yawing 0 0≤γ≤2π φ - Pitching 0 -π / 2 <φ <π / 2 ψ - Rolling 0 -π / 4 <ψ <π / 4 t _x A Tag-observation x offset 0 - t _y B Tag-observation y offset 0 - cosγ C Yawing's cosine One -1≤C≤1 sinγ D Yawing 0 -1≤D≤1 cosφ E Pitch Cosine Value One 0 <E≤1 sinφ F Sine of pitching 0 -1 <F <1 cosψ G Cosine of rolling One 0 <G≤1 sinψ H Sine of rolling 0 -1 <H <1 t _z I Tag-viewpoint z offset - I <0 1 / λ J Station of focal length - J ＞ 0 S - Viewport scale - S ＞ 0

In the x-y plane, move tx and ty according to EQ42 (when A = tx and B = ty).

Around z it is rotated by γ according to EQ43 (when C = cos (γ) and D = sin (γ)), giving EQ44.

Around x, rotate by φ according to EQ45 (when E = cos (φ) and F = sin (φ)), giving EQ46.

Around y, rotate by ψ according to EQ47 (when G = cos (ψ) and H = sin (ψ)), giving EQ48.

Perspective projection (when J = 1 / λ) in the projection plane with a focal length λ with z = 0 according to EQ53, providing EQ54.

Scale the viewport by S according to EQ55 and provide EQ56.

Move the point within the x-y plane (z = 0) according to EQ57 and give EQ58.

Finally, extend K and L and provide EQ59.

8.4.2 2D Perspective Transform

Given a 2D perspective transform matrix with an inferred eighth degree of freedom, as defined in EQ60, multiply the unknown i to obtain a matrix of the general ninth degree of freedom, as shown in EQ61.

Convert 2D points according to EQ62 and provide EQ63.

8.4.3 Inferring the Tag Transformation

8.4.3.1 Equating Coefficients

Matching the coefficients of EQ59 with the coefficients of EQ63 results in EQ64 through EQ72 with nine discontinuity equations for eleven unknowns.

These equations increase as desired by the trigonometric values associated with the angle sine and cosine values (i.e., the sine and cosine of any of yawing, pitching and rolling) as shown in EQ73.

Given the sine and cosine of the angle, the corresponding angle is obtained using the two-variable arc tangent as shown in EQ74.

8.4.3.2 Solving for X-Y Offset

EQ66 is simplified using EQ64 and EQ65 and offers EQ75 and EQ76.

EQ69 is simplified using EQ67 and EQ68, and offers EQ77 and EQ78.

The EQ72 is simplified using the EQ70 and EQ71 and offers EQ79 and EQ80.

EQ7 can be rewritten as EQ81 and EQ78 can be rewritten as EQ82.

Matching EQ81 and EQ82 and solving B yields EQ83 to EQ85 and finally EQ86, which defines B.

Substituting and simplifying the value for B into EQ82 gives EQ87-EQ90 and finally EQ91, which defines A.

This thus provides the xy offset of the tag 4 from the viewpoint, since A = t _x and B = t _y .

8.4.3.3 Solving for Pitch

EQ92 is obtained from EQ68.                 

EQ93 is obtained from EQ67.

EQ94 is obtained from EQ64, EQ92, EQ93.

EQ95 is obtained from EQ65, EQ92, EQ93.

EQ96 is obtained from EQ70, EQ92, EQ93.

EQ97 is obtained from EQ71, EQ92, EQ93.

EQ98 is obtained from EQ94.

EQ99 is obtained from EQ95.

EQ100 is obtained from EQ96.

EQ101 is obtained from EQ97.

EQ102 and EQ103 are obtained from EQ98 and EQ99.

EQ104 and EQ105 are obtained from EQ100 and EQ101.

EQ106,107 is obtained from EQ103, EQ105.

If both G and H are nonzero, only EQ107 is valid. Since | ψ | <π / 2, cos (G) of rolling is always positive and not zero. If rolling is not zero, only the sin (H) of rolling is not zero. Specific handling for zero pitching and zero rolling is described in section 6.7.3.10.

Thus F = sin (φ), this gives the magnitude of the pitch of the pitch and the cosine of the pitch according to EQ73 and EQ108.

Since | φ | <π / 2, cos (E) of pitching is always positive, therefore, there is no difficulty in taking the square root. However, the sign of sin (F) shall be determined by another method described in section 6.7.3.9.

Given E and F, pitching is obtained according to EQ109.

8.4.3.4 Solving for Roll

EQ110 is obtained from EQ103.

EQ111 and EQ112 are obtained from EQ73.

H = sin (ψ) thus gives the magnitude of the rolling sine value and the rolling cosine value according to EQ73 and EQ113.

Since | ψ | <π / 4, cos (G) of rolling is always positive, and therefore, there is no difficulty in taking the square root. However, the sign of sin (G) shall be determined by another method described in section 6.7.3.9.

Given G and H, pitching is obtained according to EQ114.

8.4.3.5 Solving for Yaw

EQ115, EQ116 are obtained from EQ73, EQ92, EQ93.

EQ117, EQ118 are obtained from EQ92, EQ116.

EQ119 and EQ120 are obtained from EQ92 and EQ116.

In EQ116, EQ118, and EQ120, the sign of the square root is determined according to the sign of i, i is determined from EQ80 and provides EQ121.                 

Since I (t _z ) is negative, J (1 / λ) is positive and IJ <−1 (since | t _z |> λ), so EQ122 is maintained.

Given C and D, yawing is obtained according to EQ123.

8.4.3.6 Solving for Viewport Scale

Yawing's cos (C) and sin (D) never become zero by definition. Since pitching cos (E) is not zero, either EQ67 or EQ68 can always be used to determine the viewport scale (S).

If D is not zero, EQ124 can be obtained from EQ67.

Otherwise, if C is not 0, EQ125 can be obtained from EQ68.

8.4.3.7 Solving for Focal Length

Similarly, rolling cos (G) never goes to zero, so if either pitching or rolling is nonzero, either EQ70 or EQ71 will be used to determine the inverse focal length (J). Can be. However, the sign of the sine values F and H of pitching and rolling may not be known. However, the sign of the product FH of the pitching and rolling sine values is given by EQ103 as shown in EQ126.

Since the sign of J is recognized as a priori, the sign can be arbitrarily assigned to F. If gi is not zero, EQ127 can be obtained from EQ70.                 

If hi is not zero, EQ128 can be obtained from EQ71.

In practice, the choice between using the EQ127 and using the EQ128 is based on which gi and hi have a higher value. If gi and hi are both zero, i.e. both pitching and rolling are zero, the reverse focal length can be perceived.

8.4.3.8 Solving for Z Offset

Once the inverse focal length J is known, the z offset I is obtained from the EQ80 according to the EQ129.

Also, if the inverse focal length J is unknown, i.e. both pitching and rolling are zero, the z offset I is unknown.

8.4.3.9 Determining Direction of Pitch and Roll

The sign of the product of pitching and rolling sine (FH) is given by EQ126. Since -π / 4 <ψ <π / 4, applying a rolling adjustment of + π / 4 can ensure that rolling always becomes positive without invalidating other assumptions. Once the rolling adjustment is applied, the EQ126 provides only the sign of sin (F) of pitching.

Rolling adjustment is applied as follows. The viewport scale S, inverse focal length J and z offset I are all calculated as described. The 3D transformation matrix is generated from the 2D perspective transformation matrix. The viewport scale, the focal length projection, and the inverse of the z movement are applied in the reverse order to the 2D matrix. The rolling adjustment is then applied by applying premultiplying of the π / 4 y rotation matrix to the matrix. Rolling, pitching and yawing are calculated as described. Since rolling is positive, the direction of pitching is also known. The π / 4 rolling adjustment is finally subtracted from the rolling to provide the actual rolling.

If rolling and pitching are both zero, both the focal length and z offset are unknown as described above. However, pitching and rolling are already known in this case, so there is no need to adjust the rolling.

8.4.3.10 Handling Zero Pitch and Roll

If either pitching or rolling is zero, the general solution based on EQ107 is invalid. The table in FIG. 85 is a degenerate form of EQ64-EQ71, with varying pitching and rolling when yawing is zero (or π, π / 2 (or 3π / 2) and not zero). Fig. 86 and 87 show that the pitching and / or rolling is zero, and in each case is driven by the zero value shown in the table of Fig. 85. The logic required to detect and handle each case is shown in Fig. 85. The cases of the table of Fig. 85 are labeled with case numbers in the tables of Figs.

CONCLUSION                 

The present invention has been described with reference to preferred embodiments and numerous specific variations. However, one of ordinary skill in the art will recognize that various other embodiments in particular to those described will fall within the spirit and scope of the present invention. Accordingly, it is to be understood that the present invention is not intended to be limited to the specific embodiments described in the detailed description, including documents incorporated by appropriate cross-reference. It is intended that the scope of the invention only be limited by the appended claims.

Claims (28)

Once placed or moved in relation to the surface, the orientation includes at least one of yaw, pitch, and roll of the housing in relation to the surface when detected by the sensing device. A sensing device for generating direction data indicating the direction of the sensing device in a relationship with the surface having encoded data indicating a, A housing; Direction sensing means configured to generate the direction data using at least some of the encoded data; And And communication means configured to communicate said direction data to a computer system. The method of claim 1, And motion sensing means for generating movement data when the sensing device moves in relation to the surface, the communication means communicating the movement data to the computer system. And a sensing device, configured to. The method of claim 2, The sensing device is configured to detect region identification data indicating an identity of the region using at least some of the encoded data when placement or movement is made in relation to a region of the surface. Further comprising region identity sensing means, said communication means being configured to communicate said region identifier data to said computer system. The method of claim 3, wherein And the movement detecting means is configured to generate the movement data using at least some of the encoded data. The method of claim 4, wherein The coded data also points to a plurality of reference points of the area, and wherein the movement detecting means generates the movement data based on movement of the sensing device in relation to at least one of the reference points. Sensing device, characterized in that configured. The method of claim 4, wherein The coded data includes a periodic element, and the movement sensing means is configured to generate the movement data based on movement of the sensing device in relation to at least one of the periodic elements. Sensing device. The method according to claim 5 or 6, And the movement detecting means is configured to generate the movement data by sampling the position of the sensing device in relation to at least one of the reference point or the periodic element. The method of claim 7, wherein And distance estimation means for estimating the distance of the sensing device from at least one of the reference point or the periodic element. The method of claim 8, And said communication means is configured to communicate distance data indicative of said distance to said computer system. The method of claim 8, The movement detecting means is configured to determine the position of the sensing device more accurate than indicated by at least one of the reference point or the periodic element by using the distance estimated by the distance estimating means. Sensing device. 5. The sensing device of claim 4, further comprising orientation sensing means configured to sense the orientation of the sensing device in relation to at least some of the encoded data. The method of claim 11, The communication means is configured to communicate orientation data indicating the direction to the computer system. The method of claim 2, The movement sensing means comprises at least one acceleration sensing means, wherein the acceleration sensing means detects the acceleration of the sensing device when the sensing device moves in relation to the surface area. And the movement sensing device is configured to generate the movement data by periodically sampling the acceleration. The method of claim 13, Said acceleration sensing means being configured to sense at least two substantially orthogonal acceleration components. The method of claim 3, wherein And a timer means configured to generate a time reference when the sensing device moves in relation to the surface area. The method of claim 15, The communication means is configured to communicate time reference data indicative of the time reference of the movement data generated by the timer means to the computer system. The method of claim 1, Said communication means being wireless communications means. The method of claim 1, And sensing force means configured to sense the force applied to the surface by the sensing device. The method of claim 18, The communication means is configured to communicate pressure data indicative of the pressure to the computer system. The method of claim 18, Further comprising stroke detection means configured to identify the duration of a stroke by detecting with said pressure when said sensing device is applied to and removed from said surface. Sensing device characterized in. The method according to claim 3, 4 or 13, And a writing nib for writing on the surface. The method of claim 21, The sensing device is in the form of a stylus or a pen. The method of claim 1, And said encoded data is substantially invisible to the unaided human eye. The method of claim 23, wherein The encoded data is printed using infrared ink, and the sensing device is responsive to the spectrum of the infrared light. The method of claim 5, The encoded data includes a plurality of tags, each tag indicating an identifier of an area in which the tag exists and an identifier of a reference point of the area, the area associated with the surface, the reference And a point indicates the location of the tag within the area. The method of claim 6, The coded data includes a plurality of tags, each tag indicating an identifier of a region in which the tag exists, and each tag includes at least one periodic element of the coded data. Sensing device. The method of claim 1, And said direction detecting means is configured to infer said direction from at least some perspective distortion of said encoded data. delete

Images