JP2008519325A - Method and device for data management in an electronic pen - Google Patents

Abstract

  A data management device in the electronic pen device operates to receive images of the product surface (602) and extract data from these images (604). The product surface may be provided with a portion of a position encoding pattern, and the extracted data may represent the position encoded by the pattern. The device selectively processes the extracted data by online processing and offline processing. Processing selection (606) may be controlled by pattern data in the image. The online processing is operable to provide 614 the extracted data to the pen device communication interface continuously and typically in near real time. The offline process is operable to store (608) the extracted data in a storage unit (M) inside the pen device. Offline processing also selectively provides the extracted data to the communication interface in a non-contiguous order (610) by retrieving and aligning the extracted data arising from a particular portion of the abstract position-coding pattern. ) Is also operable.

Description

Cross-reference of related applications

  This application includes US Provisional Patent Application No. 60 / 625,167 filed on November 5, 2004, Swedish Patent Application No. 04027710-8 filed on November 5, 2004, and June 2005. Claims the benefit of US Provisional Patent Application No. 60 / 687,332, filed on June 6, and Swedish Patent Application No. 0501265-3, filed on June 2, 2005, all such applications being , Incorporated herein by reference.

  The present invention generally relates to management of digitally recorded data, and more particularly to data management processing in an electronic pen.

  The electronic pen can be used to generate information that electronically represents handwritten writing on the product surface. One type of electronic pen works by capturing an image of the position code on the product surface. The pen can electronically record a series of positions (pen strokes) that reflect the movement of the pen on the product surface based on the image.

  International Publication No. WO 00/72230 discloses such an electronic pen, which sends the electronic pen strokes one by one to a nearby printer in near real time. The printer relays the pen stroke to a network server that provides a dedicated service. The service processes the pen stroke and causes the printer to print a response sheet, possibly position encoded. When the pen is out of range of the printer, the pen buffers the pen stroke in internal memory. When the pen is back in range of the printer, the pen transfers the buffered pen stroke in the same format as the pen stroke sent directly to the printer. The pen is designed to operate in online communication with a printer / network server. Since the pen always outputs the pen strokes continuously, i.e. in the exact order in which the pen strokes were created, the pen stroke transfer process is not visible to the user. At the same time, this principle of operation limits the usefulness of the pen.

  It is an object of the present invention to provide an electronic pen that has improved versatility over prior art pens.

  Overall, the objects of the present invention are achieved at least in part by a method and device in an electronic pen apparatus as set forth in the independent claims, an electronic pen apparatus, and a system for exchanging information with an encoded product surface. Preferred embodiments are defined in the dependent claims.

  A first aspect of the present invention is a method for an electronic pen device, which receives an image of a product surface, extracts data from the image, and converts the extracted data into online processing and offline processing. Selectively processing, the online processing is operable to continuously provide the extracted data to the communication interface of the pen device, and the offline processing is extracted. And operable to store the extracted data in a discontinuous order selectively to the communication interface. It is a method.

  According to a second aspect of the present invention, there is provided a device in an electronic pen device, an extraction processor for extracting data from an image of a product surface captured by an image sensor of the pen device; A data processor for selectively processing in off-line processing, wherein the data processor is operable to provide the extracted data continuously at the communication interface of the pen device in the on-line processing. And the data processor is operable to store the extracted data in a storage unit inside the pen device in the offline processing, and selectively selects the extracted data in a discontinuous order. Is operable to provide at the communication interface It is a vice.

  A third aspect of the present invention is an electronic pen apparatus including an image sensor for capturing an image of a product surface, a communication interface, and a processing device that executes the method of the first aspect.

  A fourth aspect of the present invention is a system for exchanging information with an encoded product surface, and establishes a connection of the electronic pen device of the third aspect to the communication interface of the electronic pen. A communication interface operable as described above, a control unit, and a feedback device, wherein the control unit provides the online data from the pen device via the communication interface when the electronic pen device provides online data. A system operable to provide feedback data via the feedback device essentially in real time with the receipt of the online data.

  Further objects, features, aspects and advantages of the present invention will become apparent from the following detailed disclosure, the appended dependent claims and the drawings.

  The invention will now be described in more detail with reference to the accompanying schematic drawings.

(General)
FIG. 6 shows a main overview of processing in the electronic pen according to the embodiment of the present invention. The pen captures an image of the surface with the coding pattern at 602. The images are processed at 604 to generate a series of data items, typically one location for each image. These positions are then continuously input to offline processing, online processing, or both based on selection mechanism 606. In offline processing, the data items are stored at 608 in the pen's fixed memory M. And later, typically initiated by the pen user, the memory M is accessed at 610 based on the selection criteria, and the resulting position is output from the pen. The selection criteria typically indicates a position due to a particular part of the coding pattern. For online processing, the position may be buffered in temporary memory B at 612 before being output from the pen at 614, at least while the pen is attempting to connect to an external device. The online processing operates to output the position at 614 continuously and in essentially real time with the image processing 604.

  Offline processing allows a pen user to create a set of electronic documents, each with a pen stroke generated by the pen, independent of the external device. The user can then cause one or more selected documents, or portions of the document, to be output from the pen regardless of the particular order in which the pen strokes were generated by the pen.

  Online processing, on the other hand, allows the pen user to generate pen strokes with interactive feedback being provided from the receiving device to the pen user via a local peripheral device such as a display or speaker or the pen itself. You can exchange information directly with the pen stroke.

  Providing both offline and online activation processing in the same electronic pen increases the versatility of the pen. For one, the user experience may be improved. This is because the pen user can now implement a very powerful new method for generating handwritten data and exchanging information.

(Abstract pattern)
The following description is based on the use of abstract position coding patterns subdivided into pages. Examples of such abstract patterns are given in US 6,570,104, US 6,663,008, and US 6,667,695, which are incorporated herein by reference. It shall be assumed. The page unit can be individually addressed in the hierarchy of the page unit group. An example is shown in FIG. In the figure, the pattern 106 includes a “segment” 110, and the “segment” 110 is divided into several “shelf” 111. Each “shelf” 111 includes several “books” 112, and each “book” 112 is divided into several page units 113. The page unit 113 is also called a “pattern page”. Preferably, all pattern pages have the same format within one stage of the pattern hierarchy. For example, some shelves may be configured with A4 format pattern pages and other shelves may be configured with A5 format pattern pages. The location of a pattern page in the abstract pattern can be represented as a page address in the form of “segment.shelf.book.page”, which is somewhat similar to an IP address, for example “99.5000.1.1500”. For reasons of processing efficiency, the internal representation of the page address may be different and may be given as an integer of a predetermined length, for example 64 bits.

In one example, each segment consists of over 26 million pattern pages, each approximately 50 × 50 cm ^{2 in} size. In one embodiment, at least one of such segments is divided into 5175 shelves, each shelf is composed of two books, and each book has 2517 pages.

  Each pattern page may be considered a de facto subset of the coding pattern, or may be considered an absolute position encoded by the subset. Each such absolute position may be represented as a global position in the coordinate system 114 of the entire pattern, or as a logical position, i.e., a page address and a local position in the coordinate system 115 within the pattern page. May be represented.

  A preferred electronic pen can perform its operation on a position-coded product, a series of global positions (ie, global pen strokes), or a series of local positions (ie, addressing) on a page address and corresponding pattern page. Recorded pen strokes).

  The following description is also based on each product that includes a position code corresponding to one or more pattern pages. However, it will be noted that the position code on the product need not be the same as the pattern page. Accordingly, one or more subsets from one or more pattern pages may be arbitrarily arranged on the product. The product may also have an embedding function in that a position code on the product is associated with one or more functions that act selectively for an electronic pen stroke that includes a position.

  In such embodiments, pen functionality may be controlled at least in part by a user operating the pen on a particular portion of the position-coding pattern. For this purpose, the pen may store one or more templates. This template defines how the pen operates on information recorded from different parts (functional areas) of the position-coding pattern. In certain embodiments, a particular page unit group in a page hierarchy, such as a segment or shelf, is associated with a given template definition, so that the template definition is all of the pages within that particular page unit group. It is valid for the pattern page. The template defines the size, placement, and function of any functional area that can affect the work of the pen. Such functional areas are denoted as “pidgets”. FIG. 4 is a diagram for explaining a state in which the template 400 is expanded on the pattern page 402 and shows various arrangements of the pidgets 404.

  In the template, all positions not occupied by the pidget in the pattern page are defined as belonging to the drawing area. Therefore, the drawing area may be regarded as an implicit pidget. The position detected in the drawing area is interpreted by the pen as to be recorded as a pen stroke.

  The pidget may indicate a trigger function, a service selection function, a device selection function, a local operation function, or a feedback function. The trigger function triggers the pen to present data as described further below. The service selection function specifies a service. The service may provide contextual information that affects pen processing for recorded positions (eg, email, fax, SMS) and / or content information that affects pen selection for positions to be processed (eg, Pages, books, shelves). The device selection function identifies a pen connection device (eg, PC, mobile device, LAN access point). The local motion function initiates an operation that affects the pen memory. The feedback function causes the pen to activate an internal MMI (Man Machine Interface) such as a vibrator, display, or speaker.

  In one embodiment, the template defines each pidget 404 with a pidget ID and a size and placement at a local location within the pattern page 402 (FIG. 4). The pidget ID is a 16-bit identifier that determines the pidget function. The pidget ID is composed of a pidget type (8 bits to 15 bits) and a pidget number (0 bits to 7 bits). The type of the pidget specifies the above function, and the pidget number may specify an auxiliary function (subfunction).

  The trigger pidgets described above can be either specific or generic. A particular trigger pidget has a pidget ID that identifies both a particular connected device and a particular service via a pidget number. Whenever the pen records a position within such a specific trigger pidget, the pen can perform the presentation of data. A typical trigger pidget has a pidget ID that does not identify a particular connected device and / or a particular service. To trigger to present data, a typical trigger pidget forms a trigger pidget ID where the pen records the corresponding position to identify both valid services and valid connected devices. And must be used in combination with a device selection pidget and / or a service selection pidget.

  In certain embodiments, a particular page unit group in a page hierarchy, such as a segment, is associated with one or more functional attributes, whereby a functional attribute is associated with that particular page unit group. It is valid for all of the pattern pages. One such attribute is the streaming attribute, which indicates to the pen that the recorded position within the page unit group should be output to the external device in real time. . The storage prohibition (DO_NOT_STORE) attribute of the page unit group prevents the pen from storing the pen strokes recorded in the page unit group. Other functional attributes may be used to categorize page units into different types, such as forms (FORMS) and notepads (NOTEPADS). In general, functional attributes may be set for any page unit group in the page hierarchy, providing the advantage of separating the generic logic from the logic provided by the template. Thus, functional attributes provide a second stage of logic. This second stage of logic can be set independently of the template logic and is available to the pen control system for the entire range of pattern pages.

(Electronic pen)
FIG. 2 is a diagram illustrating an embodiment of the pen 200 described above. The pen 200 has a pen-shaped case or outer shell 202. This case or shell 202 defines a window or opening 204 through which an image is recorded. The case includes a camera system, an electronic system, and a power source.

  The camera system 206 includes at least one illumination light source, a lens array, and an optical image reading unit (not shown). The light source is preferably a light emitting diode (LED) or a laser diode, but illuminates a portion of the area visible through the window 204 with infrared light. The image of the visible area is projected on the image reading unit by the lens arrangement. The image reading unit may be a two-dimensional CCD or CMOS detector. The CCD or CMOS detector is triggered to capture an image at a fixed or varying rate, typically approximately 70-100 Hz.

  The power source for the sensor device is advantageously a battery 208, but the battery 208 can instead be replaced or supplemented by a main power source (not shown).

  The electronic system includes a control unit 210 connected to the memory block 212. The control unit 210 is responsible for various functions in the electronic pen and may be a commercially available microprocessor such as a CPU (“Central Processing Unit”), a DSP (“Digital Signal Processor”), or an FPGA (“Field A programmable gate array ") or ASIC (" application specific integrated circuit "), discrete analog and digital components, or other programmable logic devices such as suitable combinations thereof, . The memory block 212 preferably includes various types of memory such as working memory (eg, RAM) and memory for storing program codes and fixed memory (nonvolatile memory, eg, flash memory). Associated software is stored in memory block 212 and executed by control unit 210 to provide a pen control system for operation of the electronic pen.

  The case 202 also holds a nib 214 so that the user can physically write or draw on the surface with pigment-based marking ink stored in the nib. The marking ink on the nib 214 is preferably transparent to the irradiation light in order to avoid interference with optoelectronic detection in the electronic pen. A contact sensor 216 is operatively connected to the nib 214 to detect when the pen is applied to the surface (pen down) and / or when it is lifted from the surface (pen up), In addition, the applied force can be obtained at will. The camera system 206 is controlled to capture an image between pen down and pen up based on the output of the contact sensor 216. These images are processed by the control unit 210 to generate a series of positions that represent the absolute position and movement of the pen on the encoded product.

  The generated location can be output by the pen via a built-in communication interface 218 to a nearby or remote device such as a computer, mobile phone, PDA, network server, etc. For this purpose, the communication interface 218 is a component for wired or wireless short-range communication (eg USB, RS232, wireless transmission, infrared transmission, ultrasonic transmission, inductive coupling, etc.) and / or typically May provide components for wired or wireless telecommunications over a computer, telephone, or satellite communications network.

  The pen may also include an MMI (Man Machine Interface) 220 that is selectively activated for user feedback. The MMI may include a display, a display lamp, a vibrator, a speaker, and the like.

  In addition, the pen may include one or more buttons 222 to allow for pen activation and / or control.

(Offline system)
In offline processing (offline mode), the pen may be operated to output offline data, which selectively retrieves pen stroke data from its memory block and presents this data to an external device. This is done by embedding it in a file object. Specifically, when presenting offline data, the pen stroke data stored in the memory block 212 (FIG. 2) is spatially aligned. Typically, the pen stroke data is aligned with the page address. Each file object may include pen stroke data from one or more specific pattern pages. Normally, offline data does not represent the chronological order in which pen strokes were recorded by the pen, but rather is a collection of all pen stroke data recorded in a particular portion of the position-coding pattern. In the file object, the pen stroke data may or may not be arranged in time series for each pattern page.

  FIG. 3A is an overview of an information management system that can receive offline data from an electronic pen.

  In some embodiments, the file object 300 is pushed from the pen 200 to the communication interface 301 on a given local or remote receiving device 302, 304, as shown by paths 300A-300C in FIG. 3A. Is done. The route A is set directly between the pen 200 and the remote device 304 via the communication network 310. The path 300B is set from the pen 200 through the local device 302 to the remote device 304 via the network 310. The path 300 </ b> C is set between the pen 200 and the local device 302. In another embodiment, the file object 300 is presented to be pulled from the pen 200 by the receiving device 302, 304 along any of the paths 300A-300C. In yet another embodiment, the pen 200 does not generate a file object. Instead, the receiving device 302, 304 can communicate a data request to the pen 200. After receiving the request data from the pen 200, the devices 302, 304 generate a file object 300 including this data or data derived from this data.

  The local or remote receiving device may include a flow control unit 306, which, as illustrated by paths 300C1 and 300C2, sends the fill object 300 or data extracted therefrom to the receiving device. Relay to an application program 308 that either resides locally or resides on a remote device. Examples of such flow control units are given in US 2003/0046256, International Publication WO 03/056420, and International Publication WO 03/105064. One or more flow control units may be involved in data transfer from the pen to the application program.

  The flow control unit 306 may be configured to extract relay data from the file object 300 and relay appropriate data to a destination address. This relay may be performed by, for example, HTTP (hypertext transfer protocol) post, SMTP (simple mail). Transfer protocol), X. 400, MMS (Multimedia Messaging Service), SMS (Short Message Service), database synchronization, etc. The destination address may be given directly or indirectly by the relay data. In some embodiments, the flow control unit 306 may thus extract an explicit destination address from the file object. In another embodiment, the flow control unit 306 may include a lookup table that associates address indicators with destination addresses. For example, the address indicator may be a page address, a global position, a pidget ID, a pen resident parameter value, or the like. The appropriate data to be relayed may be provided by the file object. Instead of relaying the file object 300, the flow control unit 306 may send a notification that the file object 300 can be taken out to the destination address.

  When application program 308 receives file object 300 or data derived therefrom, it may access a page description that defines the underlying physical product. The original physical product is a product from which position data in the file object 300 is generated (indicated by reference symbol P in FIG. 3A). The page description defines the layout of the original product P. This layout may include both a physical layout and a functional layout of the product. The physical layout includes supporting graphics. Assistive graphics are information that can be understood by humans, i.e. intended to instruct the user, control the user and / or inform the user. The functional layout defines the placement and size of so-called user areas on the associated pattern page (or associated pattern pages). The user area is a function area specific to an application, and has an arrangement, size, and function that may be specific to a specific application program 308. In the page description, each user area may be assigned a dedicated processing instruction identifier. The identifier of the dedicated processing instruction can access the application program. Alternatively, each processing instruction may be included in the page description. The physical layout may also be included in the page description as either an explicit instruction for generation of the physical layout or an indicator of the network storage address of the computer file that defines the physical layout. . The page description may also define the arrangement and placement of patterns on the product so that the processing application can display the pen strokes in the correct positional relationship to the supporting graphics.

  The structure, function, and use of page descriptions are further described in US 2002/0040816 and International Publication No. WO 03/105064.

  FIG. 4 further illustrates the interrelationship between the pattern page 402, the template 400, the page description 410, and the physical product 420. The physical product 420 includes a position code (not shown) that defines a position within one or more pattern pages 402 (only one is shown in FIG. 4). These locations are associated with one or more functions, which are either predefined in the system (for pidget 404) or application specific (for user area 414). is there. As provided by the template 400, the pidget 404 has a predefined arrangement and size within the pattern page 402. However, a portion of the position-coding pattern can be “cut out” and reassembled on the product 420 in any manner to cause the pidget 404 to have any arrangement on the physical product 420. May be. The page description 410 defines the arrangement, size, and function of the additional user area 414 on the pattern page 402. Thus, apart from its physical layout, product 420 has one functional layout portion 400 ′ provided by template 400 and one functional layout portion 410 ′ provided by page description 410.

  The pen stroke data in the file object 300 (FIG. 3A) is self-supporting or autonomous. That is, the receiving application program 308 can access and process data without having to communicate with the pen 200 that created the data.

The file object 300 may be generated with the pen 200 at different levels of refinement. For example, the file object 300 may include any of the following or a combination thereof.
(1) Raw pen stroke data. That is, a global position or logical position grouped together in a pen stroke.
(2) Pen stroke picture. That is, an image that directly represents the recorded position.
(3) Finely differentiated pen stroke data. Typically, information about the template such as the identification of pidgets touched by a pen stroke and / or the result of a function associated with such pidgets.

  The file object 300 may include additional data relating to the pen or additional data relating to the strokes, at any finely differentiated level. This additional data includes the time stamp of the position in the pen stroke and the time stamp of the pidget, the force value of the position in the pen stroke, the pen resident parameter (or multiple pen resident parameters). With a time stamp, an application program or flow controller could combine strokes and / or pidgets based on each other's recording time. Depending on the force value, the application program will be able to draw an image of the stroke with a line thickness corresponding to the force.

  The pen resident parameters are pre-stored in memory block 212 (FIG. 2) to identify the characteristics of the pen itself or the characteristics of the pen owner / user. These parameters may be used by the receiving application program 308 (FIG. 3A) and thus may be at least selectively included in the file object 300. The inclusion of the pen resident parameter may be determined by the page address (or page addresses) of the stroke in the file object and / or by the context information or content information described above. Pen resident parameters may include a pen's unique identifier, language identifier, name, street address, email address, telephone number, wireless caller number, fax number, credit card number, and the like.

  The file object 300 may also include independent data that is not related to the pen stroke recorded by the pen or the pen itself. Independent data may include any type of non-position data, which may be obtained by a pen sensor (eg, camera system 206) or received by the pen via communication interface 218 (FIG. 2). May be. Such independent data may be in the form of one or more images (i.e., still or moving images) or decoded from the format of data extracted from such images (e.g., a barcode or the like). Or machine-encoded text resulting from optical character recognition (OCR) processing. In yet another example, the independent data may be sound data recorded by a pen sound sensor or sound data received via a communication interface.

  File object 300 may be either character encoded or non-character encoded (eg, binary) and may be in any suitable format. The file format may be unique or standardized, and may be, for example, XML (Extensible Markup Language), HTML (Hypertext Markup Language), XForms, PDF (Portable Document Format), or the like.

(Online system)
In online processing (online mode), the pen is operated to stream data to the receiving device. This data is denoted as online data in the following. Online data may be received by an application program that implements an interactive service. A system for such interactive use of online data is shown in FIG. 3B.

  In FIG. 3B, pen 200 sends online data to communication interface 301 'on local receiving device 302' via short-range communication. Online data is transferred to a dedicated application program 308 ', optionally through a flow control unit 306' of the type described above. A dedicated application program 308 ′ is typically executed locally by a processing unit (CPU) at the receiving device, but instead is processed by a remote device 304 ′ that may be accessed via the communication network 310 ′. It may be executed by a unit (CPU). The application program 308 ′ processes the data based on the page description described above to generate feedback data, which is presented to the user in near real time via a local feedback device 312 ′ such as a display screen or speaker. (Route 300A ′ to 300C ′). Thus, the application program 308 'allows the user to access the service in near real time. The user may input handwritten data into the service and / or may control the service interactively by operating the pen 200 on the encoded product P. Further, the service may continuously display handwritten data on the screen as received from the pen, and reproduce the operation of the pen on the encoded product almost in real time.

  In the preferred embodiment described in more detail below, the pen operates in offline mode by default and activates online mode only when it records a position from the segment associated with the streaming attribute (online segment). The pen stops online mode whenever it records a position from a segment that is not associated with a streaming attribute.

  In another embodiment, the pen automatically activates the online mode when operated to present offline data (file object). This embodiment allows the user to generate handwritten data offline and interactively against the same handwritten data when this data is deemed received by an external device application program. Can be operated. For example, the user may control the display of handwritten data by zooming or scrolling by operating a pen on the original product. In another example, an application program can perform HWR processing on handwritten data so that the user can interactively correct, annotate, or edit handwritten input by operating the pen on the underlying product. You may do it. The application program may automatically manipulate the online data for the last received offline data. Alternatively, the online data may be used by an application program to retrieve the associated offline data from memory accessible to the external device, where the application program manipulates the online data against this offline data. For example, the application program may use the page address of online data to retrieve offline data having the same page address.

  Pen operation in the online mode may be controlled in part by pidgets as in the offline mode. However, in the preferred embodiment, the online segment template has no pidget other than the drawing area. Alternatively, the pen may be designed to ignore such pidgets in online mode. To implement pen MMI feedback, the pen may be designed to selectively activate its MMI 220 (FIG. 2) based on feedback instructions. This feedback command is received at the communication interface 218 from the application program. Thereby, all user feedback in the online mode is controlled by the application program.

  The pen may send online data to a receiving device that is currently connected to the pen. Alternatively, the pen may initiate a connection to the default online data receiving device or the receiving device to which the pen was last connected in online mode. In another alternative approach, the receiving device may be provided by the pen detecting a dedicated pidget or by the user physically connecting the pen to the receiving device. It is also conceivable that the connection is started from the receiving device.

(Pen control system)
The pen 200 operates by executing software in the control unit 210 (FIG. 2). The overall software architecture will now be described with reference to FIG. 5, which illustrates several main software components.

  The pen system software is based on modules. Modules are independent entities in the software with a clean interface. A module is either active by including at least one process or passive by not including a process. The module may have a function interface that performs function calls or a message interface that receives messages. An active module and a passive module are basically organized as a tree where the module's parent is responsible for starting and shutting down all its children.

  The pen system software also implements an event framework to reduce dependencies between modules. Each module may present a predefined set of events that it can signal. In order to receive notification of a specific event, the module must be registered in the event register for this event. The event register may also indicate whether notification is to be made by posting a message or as a callback function.

  The combination of software modules implements a so-called coordinate pipeline 502, which is a flow of events for storing strokes (offline mode) or for streaming positions (online mode). . The global position stream from the image processing module 504 is transmitted to the conversion unit module 506, and the conversion unit module 506 converts the global position into a logical position. When the converter module 506 detects that a stroke passes through a pidget, the corresponding pidget event is triggered. Also, if the global location segment is associated with a functional attribute, the transform module 506 also generates a specific attribute event. Appropriate operations may be performed in other software modules by pidget and attribute events, which include an event listener that determines the validity of each pidget and attribute event. Unless a storage prohibit attribute event is generated, the logical position is received by the coordinate manager module 508, which coordinates the logical position into addressed pen strokes, which are concise representations. Are stored in the memory block 212 (FIG. 2).

  When the streaming attribute event is generated, the logical position is also received by the stream module 508 ′, which generates online data, while the presentation module 516 is online connection to the receiving device. Is established.

  Thus, particularly in this embodiment, the pen may operate exclusively in offline mode (by default) or may operate exclusively in online mode (the drawn pattern may be streamed attributes and stored). It may operate in both offline and online modes (if it is associated with both forbidden attributes) (the drawn pattern is associated with streaming attributes but not with storage prohibited attributes) If not).

1. Supervisory Unit As shown in FIG. 5, the pen system software includes a supervisory module 509 that initiates all other modules directly or indirectly. Since the supervisor module 509 includes an interface to the event register, the module can register itself for the event it wants to receive. The supervisor module 509 also generates control events that can be used by listeners of other software modules.

2. A server module 510 is provided as an event coordinator for the server pen. Server module 510 determines what the pen should do based on the status of the system and incoming events.

3. Image Processing Unit The image processing module 504 will not be further described here. The image processing module 504 includes, for example, US 2003/0053699, US 2003/0189664, US 2003/0118233, US 2002/0044138, US 6,667, previously published by the present applicant. 695, US 6,732,927, US 6,929,183, US 2003/0128194, and references therein.

4). Location Database The pen includes a location database 511 that resides in the pen's memory block 212 (FIG. 2) and can be edited or updated via the communication interface 218 (FIG. 2). The position database 511 is used by the conversion module 506 and includes a segment database 511A. This segment database 511A associates each segment with a template given by a template index and optionally also associates with one or more functional attributes. The segment database 511A is preferably cached in RAM memory for fast access. The location database 511 also includes a template database 511B that associates the template index with a pointer to the template definition. The template definition may be stored as properties in the pen's non-volatile memory. Such a template definition includes a page X dimension (16 bits), a page Y dimension (16 bits), a page number (16 bits), a pidget number (16 bits), and a pidget list (length according to the pidget number). May include. The pidget list is composed of several pidgets. Each pidget has a pidget ID (16 bits, upper 8 bits are type, lower 8 bits are numbers), upper left X coordinate (16 bits), upper left Y coordinate (16 bits), width (16 bits), height (16 Bit).

5. Conversion Unit The conversion unit module 506 uses an algorithm based on known pattern formatting data in order to convert a global position into a logical position. The pattern formatting data defines the hierarchical division of the encoded pattern into pattern pages (FIG. 1). Some of the pattern formatting data may be stored in the pen in advance, and other parts may be dynamically derived from the pen position database 511. For example, if all segments have a certain size, the global location may be processed to identify the current segment of this global location. Once the current segment is known, the segment's template definition in the location database 511 may be used to derive data about the segment split, where the current shelf, book, page, and local location, i.e. the complete logical location. The global position may be further processed to identify. If the pen is normally operated on the same pattern page for a while, the current pattern page boundary is cached at the global location, and the new global location is checked against this boundary Speedup may be achieved. As long as the new global location is within the boundary, there is no need to recalculate the page address.

  Also, the converter module 506 detects whether the recorded position is in the pidget on the current pattern page by mapping the local position to the associated template definition. This template definition is given by the location database 511. In some embodiments, only one pidget can be identified for each stroke. That is, if several pidgets are identified, only the last pidget is considered. A corresponding pidget event is then generated for notification to other software modules. If no pidget is specified, the position is assumed to be in the drawing area.

  The converter module 506 also detects whether the current segment is associated with a functional attribute. Then, in order to show each such functional attribute to other software modules, a corresponding attribute event is generated and notified.

  Thus, the transform module 506 provides at least a page address and a local location for each global location, and may also provide an attribute indicator. The converter module 506 may also provide a pidget ID for each of a group of temporally aligned positions, ie, strokes. Each logical position is associated with a time stamp provided by the time circuit of control unit 210 (FIG. 2) and a force value provided by contact sensor 216. Similarly, each pidget ID is associated with a time stamp. Further, the transform module 506 may have an interface that allows other modules to look up information in the location database 511 for templates, functional attributes, and pidgets.

6). Coordinate Management Unit The coordinate management unit module 508 receives a logical position from the conversion unit module 506. The coordinate manager module 508 groups the logical positions into temporal arrangements, ie, strokes, prior to storage. The coordinate manager module 508 may preprocess each stroke for compression and store the result in a non-volatile memory. Examples of such compression and storage are given in US 2003/0123745 and US 2003/0122802.

  The coordinate manager module 508 also includes an interface for other modules to search for stored strokes and retrieve strokes in an off-line transport format, eg, based on page addresses. In one embodiment, the offline transfer format is binary and includes the following data: the start time of each stroke, the local position at each stroke, and the force value at each position.

  Optionally, the off-line transfer format may also include orientation data derived from the captured image to indicate the three-dimensional orientation of the pen during position recording.

7). Stream Part The stream part module 508 ′ may be activated to receive a logical position from the converter module 506 and has three internal states. The three internal states are disconnection, connection establishment, and connection. The stream module 508 ′ inputs the respective states based on the connection establishment event, the connection event, and the connection failure event generated by the presentation module 516 (see below).

  In the disconnected state, the pen system software is in offline mode. In the connection establishment state, the stream unit module 508 ′ displays all the logical positions output from the conversion unit module 506 (along with the force value and time stamp), and the buffer memory (typically In other words, it is continuously stored in a RAM). The duration of the connection establishment state is typically about 1 to 10 seconds. In the connected state, the stream module 508 'generates online data according to a predetermined binary online transfer format. In one embodiment, this format includes three different messages:

New session (time stamp, pen identifier)
New position (time stamp, page address, position, force value)
Pen up (time stamp)

  When the stream module 508 'detects a connection event, it generates a new session message and includes a time stamp representing the time when the online connection was established. The stream module 508 'then generates new position messages and includes one logical position, force value, and time value in each new position message. The time value represents the time when the original image was captured by the pen camera system. Each new position message may also include the aforementioned azimuth data. As indicated by the touch sensor 216 (FIG. 2), the stream module 508 'generates a pen up message whenever the pen is moved away from the writing surface.

  In an alternative embodiment, the page address is output only once for each pen stroke. For example, as described in the aforementioned US 2003/0123745 and US 2003/0122802, local positions are excluded from each pen stroke according to a resampling criterion to further reduce the amount of data transferred. And / or each local position may be given as a difference value relative to a preceding local position within the same stroke.

  The stream module 508 ′ always processes the logical positions in the order generated by the image processing module 504. Thus, the stream module 508 ′ first retrieves and processes the position stored in the buffer memory during the connection establishment state, and then processes the generated position, which is necessary. For example, it may be performed via an intermediate storage unit in the buffer memory.

  If the pen control system is instructed to deactivate or switch to offline mode, the stream module 508 ′ stays connected until it has processed all the data in the buffer memory, thereby providing the presentation module 516. This data is output to the online connection. If a connection failure event is received by the stream module 508 'in the connection establishment state, the stream module operates to delete all data from the buffer memory.

  The online transport format allows the receiving device to distinguish the data generated during the connection establishment state and the connection state, respectively. The location timestamp recorded during the connection establishment state will precede the new session message timestamp, whereas the location timestamp recorded during the connection state will be It will follow the timestamp. Alternatively or additionally, each new position message may include a bit value to indicate whether the data is buffered or not.

8). Parameter Storage Unit The pen system also includes a parameter storage module 512. All other modules that need to maintain state after the pen is deactivated call this module to store their state parameters. The parameter storage module 512 handles not only the template definition but also pen resident parameters set at the time of manufacture such as unique pen properties (such as pen identifiers) or calibration parameters. Each parameter is stored in a non-volatile memory along with a name, size, and data. Module 512 provides an interface 512 ′ for writing, retrieving, and deleting parameters.

9. File Assembly A file assembly module 514 is implemented to generate a file object 300 (FIG. 3A) to be presented to a data handler external to the pen. Module 514 is implemented to listen for dedicated events such as trigger pidget events. When module 514 detects a dedicated event, it retrieves the stroke associated with the trigger pidget from coordinate manager module 508 in a binary offline transport format.

  Obviously, the function of the file assembly module 514 will depend on the level of how fine it differentiates. The above steps are sufficient for level 1 fineness differentiation.

  For level 2 fineness differentiation, the pen system includes a stroke converter module (not shown) that transfers a given set of strokes to an image buffer in memory block 212 (FIG. 2). Well, it may also include an image encoder that encodes the image buffer into a given type of image, for example a bitmap format or a vector graphics format.

  For level 3 granularity, module 514 may retrieve data items provided by other pidgets associated with the trigger pidget from other modules (not disclosed). Such data items include machine-encoded characters by handwriting recognition (HWR) of pen strokes, machine-encoded characters taken from pen memory, and communication addresses taken from pen memory. Etc. may be included.

  The module 514 may extract independent data to be included in the file object such as barcode data, OCR text, sound, and image.

  Finally, the module creates a file object 300 (FIG. 3A), which is made available to other modules via the interface 514 '.

  Such a file object may have the following structure, for example.

-File header (trigger pidget ID, page address, number of pages, number of additional data items)
..Page data header (page address, number of pidgets)
... Page data (stroke data in offline transfer format)
... Pidget data (Pidget ID (or multiple pidget IDs), time stamp (or multiple time stamps))
..Additional data (pen resident parameter values (or multiple pen resident parameter values), data items resulting from pidgets, independent data)

  Depending on the file object, the application program 308 (FIG. 3A) and optionally the flow control unit 306 can identify and extract all or selected portions of the file object.

  The selection of the stroke to be included in the file object may be determined from the content information included in the trigger pidget ID. Such content information may indicate, for example, the current pattern page or book. That is, the selected stroke must belong to the same set of pattern pages as the trigger pidget. It is clear that there are alternative ways of selecting a stroke. For example, a stroke may be selected from within a bounding area defined by a dedicated pen stroke (ie, the pen is moved over the product to indicate to the pen what to present). In another example, a stroke may be selected based on a classification attribute associated with a pattern page. For example, a stroke may be selected from all pattern pages associated with the aforementioned form attributes. In yet another example, all strokes in pen memory are automatically selected for presentation, preferably with one file object created for each pattern page or book.

  All strokes thus selected may be incorporated into the file object. Alternatively, only strokes that are not represented as pidgets are incorporated into the file object.

10. Presentation Unit A presentation module 516 is provided to provide the collected data to a data handler outside the pen. This module is implemented to listen for dedicated events such as trigger pidget events or streaming attribute events.

  When this module detects a trigger pidget event, it retrieves the file object from the file assembly module 514 and presents the file object. This presentation can be done, for example, either by pushing the file to a specific device / port given by the trigger pidget or device selection pidget, or by allowing an external data handler to pull the file object from the pen. Is called. For example, the file object may be stored in a file system in pen memory. This file system is presented to an external data handler. For example, the file system may be presented for viewing via a suitable protocol such as USB (Universal Serial Bus), Bluetooth (registered trademark), FTP (File Transfer Protocol), HTTP (Hypertext Transfer Protocol).

  When the presentation module 516 detects a streaming attribute event, it initiates an online connection to a particular receiving device and generates a connection establishment event. When the connection is established, the module generates a connection event and starts to continuously output messages generated by the stream module 508 '. If the connection to the receiving device cannot be made within the timeout period, the module generates a connection failure event.

  The presentation module 516 contains all the necessary protocol stacks and is designed to be independent of the transmission means, but this assigns responsibility for setting up the link to another module and works by device abstraction It is realized by. The person in charge of setting the link is assigned to a link selection module (not shown). The link selection module handles communication link selection and configuration.

  The present invention has been generally described above with reference to several embodiments. However, as will be readily appreciated by those skilled in the art, other embodiments than those disclosed above are equally possible within the scope and spirit of the invention, and the invention is not limited to the appended claims. It is defined and limited only by the scope.

  For example, an event may be generated in a pen control system by a user pressing a button on a pen or issuing a verbal command to be recorded by a pen microphone. Thus, a trigger pidget event or any functional attribute event may be replaced or supplemented with such an event.

  In the above embodiment, the abstract pattern segmentation is dynamic in that the logical position is calculated based on the pen position database. In an alternative embodiment, abstract pattern partitioning may be static by being encoded in the pattern. For example, US Pat. No. 6,330,976 discloses an encoding pattern in which encoded cells are tiled on the product surface, each cell having a local location and page identifier. And both. Such a pattern may implement functional attributes as one or more flag bits encoded in the pattern. Thus, the pen can directly detect its logical position by decoding the pattern, and can optionally detect functional attributes.

  It will also be noted that the pen may include complementary equipment for relative positioning, such as an accelerating watch, a roller ball, a triangulation device, and the like. As such, the pen may supplement the absolute position derived from the position code with the relative position provided by the complementary instrument. In this case, the position code need only encode a few absolute positions on the product.

  Furthermore, the pen system software may be based on the distribution of functionality between separate software modules. For example, the functions of several modules may be implemented by a single module, which is realized, for example, by the server module also implementing the functions of the presentation module. In fact, the pen system software does not have to be composed of separate modules.

  Similarly, event signaling may be omitted or reduced. In one such embodiment, attribute events are omitted. Instead, the converter module sends all logical positions to both the coordinate manager module and the stream module. Whenever the conversion module detects a storage prohibit attribute, it sends the associated logical location only to the stream module. The stream module continuously accesses the converter interface to check whether the received logical position is associated with the streaming attribute. When such an attribute is detected, the stream module causes the presentation module to start an online connection.

  Furthermore, the pen may output offline data in ways other than those described herein. For example, the pen may establish end-to-end communication with the receiving device and output offline data as part of an HTTP request to the receiving device. A protocol for such communication is further described in Applicant's patent publication US 2003/0055865, which is incorporated herein by reference.

FIG. 1 is a diagram for explaining logical division of an abstract position coding pattern into an addressable page unit tree structure. FIG. 2 is a cross-sectional view of an electronic pen that implements the principles of the present invention. 3A and 3B are diagrams for explaining an information management system and its communication path for generating and processing offline data and online data, respectively. FIG. 4 is a diagram for explaining the relationship between the template definition and the page description to the final position-coded product. FIG. 5 is a diagram illustrating an embodiment of a system of software modules for controlling the operation of the pen of FIG. FIG. 6 is an overview of a process for generating position data in the electronic pen according to the embodiment of the present invention.

Claims (36)

A method in an electronic pen device, comprising:
Receiving an image of the product surface;
Extracting data from the image;
Selectively processing the extracted data by online processing and offline processing,
The online processing is operable to continuously provide the extracted data to the communication interface of the pen device;
The offline processing is operable to store the extracted data in a storage unit inside the pen device, and selectively provides the extracted data to the communication interface in a discontinuous order. A method characterized by being operable.   The method of claim 1, wherein the online processing is operable to provide the extracted data at the communication interface essentially in real time with the extracting.   The extracted data includes a plurality of positions, each of the positions is associated with a position area identifier, and the extracted data in the discontinuous order is aligned based on the position area identifier. The method according to claim 1 or 2, wherein:   4. The offline processing is operable to provide the extracted data to at least one autonomous data file object suitable for storage in a computer file system. The method of crab.   5. A method as claimed in any preceding claim, wherein the offline processing is operable to selectively provide the extracted data only upon detection of a dedicated trigger event.   6. The method of claim 5, wherein the dedicated trigger event arises from the communication interface connected to an external device or from a predetermined pattern detected in at least one of the images.   The method according to claim 1, wherein at least one of the online processing and the offline processing is selected by detecting a predetermined pattern in at least one of the images. The extracted data is processed by the offline processing by default,
The method according to claim 7, wherein the data extracted following the detection by the detection is processed in the online processing.   9. The method of claim 7 or 8, further comprising generating a dedicated processing event to indicate the detection.   The method according to claim 7, wherein the online processing is started only during the detection of the predetermined pattern in the image.   The method according to claim 9 or 10, wherein the predetermined pattern is detected from the extracted data. The extracted data is a method comprising a position in a global coordinate system,
The method further comprises deriving a processing indicator by mapping at least one of the positions to definition data stored in a memory of the pen device;
The method according to claim 7, wherein the predetermined pattern is detected from the processing indicator. The definition data indicates position areas that are individually addressable in the global coordinate system, each of the position areas being associated with a position area identifier;
The method of claim 12, wherein the processing indicator comprises the location region identifier. The definition data indicates position areas that are individually addressable in the global coordinate system, and associates a set of the position areas with functional attributes;
The method of claim 12, wherein the processing indicator comprises the functional attribute.   The method of claim 7, wherein the online processing is initiated by the offline processing selectively providing the extracted data to the communication interface in a discontinuous order.   16. The off-line process is further operable to identify a function to be performed by the pen device based on the extracted data. Method.   The method of claim 16, wherein the function comprises an instruction to activate a man machine interface (MMI) of the pen device.   The online processing is further operable to receive a feedback command at the communication interface and is operable to activate an MMI of the pen device based on the feedback command. The method in any one of 1-17.   The online processing is further operable to store data extracted from an image acquired during the initiation and establishment of a connection to an external device via the communication interface in a buffer memory of the pen device. And is operable to provide, after the establishment, the data stored in the buffer memory and data extracted subsequent to the establishment to the external device via the communication interface. The method according to claim 1, wherein the method is characterized in that   The method of claim 19, wherein the online processing is further operable to erase the data stored in the buffer memory if the connection is not established.   The online processing is further operable to provide, via the communication interface, a buffer indicator that identifies the extracted data that was stored in the buffer memory prior to the establishment. The method according to claim 19 or 20. The buffer indicator comprises a capture timestamp for each data item included in the extracted data provided to the communication interface, and a connection timestamp;
The method of claim 21, wherein the capture timestamp represents a time when the data item was recorded in the image, and the connection timestamp represents a time for establishing the connection. .   23. A session message comprising a connection time stamp representing a time for establishing a connection to an external device via the communication interface, wherein the online processing provides a session message. Method.   The method of claim 23, wherein the session message further comprises a pen identifier that uniquely identifies the pen device.   25. The online processing provides the extracted data as a series of data messages, each of the data messages comprising a data item extracted from one of the images. The method in any one of.   The method of claim 25, wherein each of the data items comprises a location and a location region identifier.   27. A method according to claim 25 or 26, wherein each of the data messages further comprises a time stamp representing the time when the image was acquired.   28. A method according to any of claims 25 to 27, wherein each of the data messages comprises a force value obtained from a force sensor associated with a pen tip of the pen device.   29. A method according to any of claims 25 to 28, wherein each of the data messages comprises orientation data representing a three-dimensional orientation of the pen device, determined from the image. A device in an electronic pen device,
An extraction processor for extracting data from the product surface image captured by the image sensor of the pen device;
A data processor for selectively processing the extracted data by online processing and offline processing;
With
The data processor is operable to provide the extracted data continuously at the communication interface of the pen device during the online processing;
The data processor is operable to store the extracted data in a storage unit inside the pen device in the offline processing, and selectively extract the extracted data in a discontinuous order. A device characterized in that it is operable to provide at a communication interface. An image sensor for capturing an image of the product surface;
A communication interface;
A processing device for performing the method according to any of claims 1 to 29;
An electronic pen device comprising: A system for exchanging information with an encoded product surface,
The electronic pen device according to claim 31;
A communication interface operable to establish a connection to the communication interface of the electronic pen;
A control unit;
A feedback device;
With
The controller is operable to receive the online data from the pen device via the communication interface when the electronic pen device provides online data, essentially with the reception of the online data. A system operable to provide feedback data via the feedback device in real time.   The system of claim 32, wherein the controller is operable to retrieve offline data previously provided by the pen device when the pen device provides the online data.   The system according to claim 33, wherein the control unit specifies the offline data to be extracted based on the online data.   The system according to claim 32 or 33, wherein the control unit operates the online data with respect to the retrieved offline data.   36. The system according to claim 32, wherein the duration of the connection is controlled by the electronic pen device.