KR101045444B1 - Electronic ink processing - Google Patents

Abstract

A system, method, and computer readable medium for making rich, flexible, and flatter electronic ink annotations in an electronic document, comprise generating a first context node associated with a first portion of a base portion of the electronic document. ; Creating a second context node associated with the annotation for the reference portion; Linking the first context node and the second context node.

Electronic ink, annotations, context nodes, electronic documents, links

Description

Electronic ink processing {ELECTRONIC INK PROCESSING}

The present invention relates to the processing of electronic inks. In particular, various embodiments of the present invention are applicable to the analysis of electronic ink, including layout analysis, classification, and recognition of electronic ink. Additional embodiments of the present invention are directed to the use of electronic ink in which the layout is analyzed, classified and recognized, for example, in providing rich and flexible annotations in an electronic ink document. It is about.

As the role of computers in society increases, many different technologies have been developed for entering data into computers. One particularly useful technique for entering data is by handwriting. By writing using a stylus or other object on a digitizer to produce an "electronic ink", a computer user can be freed from the inconvenience and large volume associated with a keyboard. Handwriting input can be conveniently used by, for example, roundabout doctors, architects on a building site, delivery workers delivering parcels, and warehouse workers who walk around the warehouse, making it difficult or inconvenient to use the keyboard. It can be used in any situation. In many situations, handwriting input is more convenient than keyboard input, but typically most software applications cannot directly process text written in electronic ink. Instead, text written in electronic ink must be analyzed to be converted into another form, such as ASCII characters. This analysis involves handwriting recognition processing, which recognizes characters based on several relationships between the individual electronic ink strokes that make up a word of electronic ink.

Handwriting recognition algorithms have been tremendously improved in recent years, but their accuracy can be compromised if electronic ink is written tilted. Likewise, when separate groups of ink strokes cannot be easily distinguished, such as when two words are written together close to each other, many recognition algorithms cannot accurately recognize electronic ink. In addition, some recognition algorithms may in fact recognize the electronic ink as text even if the electronic ink is intended to draw. For example, by writing an electronic ink stroke by underlining, highlighting, or displaying circles or crosses over a portion of the typewritten text. You can annotate typed text. The handwriting recognition algorithm may misrecognize these annotation strokes as dashes, digits "0" or letters "O".

The accuracy of many recognition algorithms can be significantly improved by "parsing" the electronic ink (eg, by analyzing the layout of the electronic ink and / or by "classifying" the electronic ink) before using the handwriting recognition algorithm. Can be. Typically, the classification process determines whether the electronic ink stroke is part of a picture (ie, picture ink stroke) or part of handwritten text (ie, text ink stroke). Classification algorithms for identifying other stroke types are also possible. The layout analysis process typically groups the electronic ink strokes into meaningful combinations such as words, lines and paragraphs. Therefore, the layout analysis and classification processes include which strokes in the collection of electronic ink belong to one word, which words of the electronic ink are associated with one line of text written in electronic ink, and which of the text written in electronic ink. Can be used to identify if a line is associated with a paragraph.

Layout Analysis and Classification Inks can greatly improve the recognition of electronic ink, but many software application developers are not aware of the importance of these activities prior to electronic ink recognition. Until recently, layout and classification algorithms for use with existing software applications were not readily available. For example, the Microsoft® Windows XP Tablet PC Edition Version 2002 operating system was typically sold with a Microsoft® Windows Journal software application for storing, displaying, and processing electronic ink. The Microsoft® Windows Journal software application uses an internal parser, which until recently had no access to other software applications run by the operating system.

Although parsing processing from Windows Journal software applications is now separately accessible by other software applications, it is not well known to use such a parser, and such parsers with many software applications where the user wishes to enter handwriting inputs. Using it may not be easy. In addition, even if a software application developer wishes to create a parser specifically to use the desired software application (which can be difficult and time consuming in itself), the execution of the parsing process can be very time consuming. For example, parsing only a few strokes of electronic ink using a relatively fast microprocessor may take seconds or even minutes. If a software application needs to stop working until the parsing process is complete, the software application will be too slow for most users to use practically.

Thus, there is a need for electronic ink processing techniques that can be used by various software applications, for example, to analyze the layout of electronic ink and to classify and / or recognize electronic ink. There is also a need for an electronic ink processing technique capable of processing electronic ink and for allowing software applications using the techniques to accept new electronic ink input without invalidating the results of the ink processing.

Summary of the Invention

Advantageously, various embodiments of the present invention provide electronic ink processing techniques that can be used by various software applications to process electronic ink. In addition, these electronic ink processing techniques allow the electronic ink to be processed asynchronously with the operation of the software application implementing the techniques, so that the electronic ink can be processed without stopping or significantly delaying the operation of the software application. Software applications can continue to accept new electronic ink inputs even while previous electronic ink inputs are being processed.

In various embodiments of the present invention, elements in a file or document may be described based on their spatial location relative to each other. For example, both the electronic ink stroke and the typed text can be described by the same spatial coordinate system. Using spatial information to describe the elements of a document, the software application managing the document can maintain a data structure describing the relationships between the document elements. In particular, the software application may describe a class of various document elements and maintain a data structure that defines the associations between the various document elements. Such associations may be defined, for example, as information used to link electronic ink stroke data or sets thereof to other elements (such as words, rows, paragraphs, pictures, table cells, etc.) within an electronic document. .

By describing document elements within a file or document data structure based on their spatial location, document elements for various file types may use common techniques to identify and process those document elements. In particular, various software applications can describe document elements in a document based on their spatial location and can use this spatial location reference to use common electronic ink analysis methods. In addition, by designating specific areas of the document for analysis, each software application can limit the analysis process to only the desired elements in the document.

In order to analyze a new electronic ink input into a document in accordance with various embodiments of the present invention, the software application managing the document changes the data structure associated with the document to include the new ink to be analyzed. The software application then provides this data structure (or parts thereof) to the ink analysis tool, which copies some or all of the data structure for analysis (and separates the data from the document data structure of the application). Works on copy). The ink analysis tool delivers its copy to an analysis process, such as a parsing process (eg, layout analysis process and / or classification process). The software application may resume normal operation, including receiving new electronic ink input and / or other data while the ink analysis processing (s) is performed. In addition to receiving a new electronic ink, any " other data " such as data for changing the content, position and size of existing ink, text, images, graphics, tables, flowcharts, diagrams, etc .; Data to add additional text, images, graphics, tables, flowcharts, diagrams, and the like; Data that deletes existing text, images, graphics, tables, flowcharts, diagrams, and the like may be received by the application. After all desired analysis processes have been completed, the analysis results are returned to the ink analysis tool.

Therefore, various examples of systems and methods in accordance with the present invention allow ink analysis processes to run asynchronously with the operation of a software application that uses the ink analysis process. This asynchronous operation allows the user to continue using the software application without being delayed by the analysis process. In addition, this allows multiple analysis processes to be executed simultaneously.

In response to receiving the analysis results, the ink analysis tool obtains a current version of the data structure of the electronic document from the software application (which may include new and / or changed data entered while the analysis processes are performed), and the data Reconcile the analysis results with the current version of the structure. By adjusting the analysis results using the current version of the data structure, various embodiments of the present invention avoid more complex techniques, such as "locking", for asynchronous access to data being used by software applications. Can be. Instead, reconciliation can be invoked by any software application without the need to provide complex internal locking.

After adjusting the analysis results using the current version of the data structure, the ink analysis tool can provide a copy of the adjusted analysis results to another analysis process, such as a handwriting recognition process. Again, the software application may resume normal operation, including receiving new electronic ink input and / or other data while the second ink analysis processing (s) is being performed. After all the desired second analysis process is completed, the results of the second analysis process are returned to the ink analysis tool. The ink analysis tool then obtains a current version of the data structure from the software application (which may include new and / or modified data again) and adjusts the second analysis results using the current version of the data structure. The ink analysis tool then updates the data structure with the adjusted results of the second analysis process. Of course, any number of ink analysis procedures and / or steps may be used without departing from the present invention.

The use of spatial information that relates or links the various ink analysis processes and electronic ink data described above to other features of an electronic document can be used to provide rich, flexible, and natural ink annotations to an electronic document. For example, embodiments of the present invention may be used to provide electronic ink annotations that move and / or change dynamically based on changes made to annotated underlying document elements. Users typically have a number of different ways (e.g., circle, underline, highlight or underline words). ; You can write notes in the margin; you can annotate documents with arrows or other pointers in the comments located in the margin. In addition, users can annotate a wide variety of different document types, including, for example, text, spreadsheets, drawings, slideshows, tables, charts, graphs, flowcharts, and the like.

In order to seamlessly integrate electronic ink annotations into an electronic document, the annotations need to behave properly when the underlying electronic document changes for some reason. For example, a user marking a circle (as a comment) on a word in an electronic document and then adding text somewhere in the document, before that word, may result in the circle moving the marked word. In this case, the circle comment should move and stay with the word. As another example, if a user adds or removes letters to a word that marks a circle, or changes its size in any way, the circle annotation must expand or contract to accommodate the new size of the word. Preferably, when the underlying electronic document updates or reflows the page and / or the relative position of the constituent elements (paragraphs, pictures, columns, etc.) with respect to each other, the electronic ink annotations also include the base text or You will stay or reposition in appropriate positions with respect to other information. If the comments do not react in this way to the underlying electronic document text, then it is still not practical to edit or share the live electronic documents after commenting, so parer printout is still the easiest and useful way to annotate. Will be

Determining how to properly change annotations will depend on several factors regarding annotations and electronic documents. For example, properly changing the annotation in a reflow situation may include, for example, (a) identifying the electronic ink as tin, (b) identifying the type of electronic ink tin, and (c) the background electron. It may depend on the computing system's (eg, parser's) ability to identify the relationship of the electronic ink to a particular element in the document. It may be possible to query the user for some or all of these types of information, but such a system will have a much more troublesome experience than if the user annotates on paper. Thus, according to various embodiments of the present invention, the information can be inferred from the content of the underlying document, including (1) the electronic ink itself and (2) the spatial location of the various elements in the document with respect to the electronic ink. .

Determining the meaning of ink annotations can be complex and difficult, so it is very impractical to expect each application of a document that a user wishes to annotate to implement annotation identification logic separately. Rather, it is desirable to provide a reusable component to aid this annotation function that each application can easily integrate. For example, ink annotations that can operate very easily with associated inks, such as the operating system of a pen-based computing system providing components for the rendering and collection of electronic inks. It would be desirable for the operating system to provide a component (also referred to as an "annotation parser") that determines their meaning.

However, identifying the content of the base electronic document introduces significant difficulties in providing "smart" annotation capabilities. For example, various different applications have very heterogeneous ways of storing, managing, and reflowing documents. The present invention provides a reusable annotation parser that can determine the relationship of possible ink annotations to various different types of electronic documents. Specifically, according to some embodiments of the present invention, information relating to an electronic document in a spatial area corresponding to the ink annotation to provide relevant information about the ink being parsed (eg, to provide “context” to the ink annotation). An annotation parser is provided to the mechanism that calls back to the application to provide. The mechanism is easy enough to fit into any document-based application and is efficient enough to work with very large documents (which can be processed in sections such as pages).

Therefore, ink processing techniques in accordance with various embodiments of the present invention allow various software applications to perform multiple processing on electronic ink through an ink analysis tool. In addition, a software application using these techniques can continue normal operation during the analysis process, including receiving a new electronic ink input, without necessarily invalidating the results of the analysis processes.

These and other objects, features and advantages of the present invention will be readily apparent and fully understood from the following detailed description set forth in conjunction with the accompanying drawings.

1 is a schematic diagram of a general-purpose digital computing environment in which some embodiments of the present invention may be implemented.

2 is a schematic diagram of a pen-based personal computing (PC) environment in which some embodiments of the invention may be implemented.

3 and 4 illustrate various features of exemplary implementations of embodiments of the present invention with regard to electronic document annotations.

5 illustrates general features of ink parsing of an electronic document.

6A-6I illustrate exemplary data structures useful for practicing at least some embodiments of the present invention.

7A-12B illustrate exemplary electronic documents and related exemplary data structures useful for executing electronic ink annotations of these electronic documents.

13A-14B illustrate exemplary electronic documents and related exemplary data structures useful for processing electronic flow chart features of the electronic documents.

15A-15C illustrate exemplary electronic documents and related exemplary data structures useful for processing electronic table features of the electronic documents.

16A-16D illustrate a method of processing an electronic ink in accordance with various embodiments of the present invention.

17, 19-21, and 23-26 illustrate the transfer of data objects during an exemplary ink analysis process in accordance with various embodiments of the present invention.

18 and 22 illustrate simple data trees that can be handled in accordance with various embodiments of the present invention.

FIG. 27 is a flowchart showing a method of adjusting analysis results to a current state of a document. FIG.

FIG. 28 illustrates an apparatus for asynchronously analyzing electronic ink in accordance with still other examples of the present invention. FIG.

DETAILED DESCRIPTION OF THE DRAWINGS

<Terms>

The following terms are used herein, and the terms have the meanings provided below unless otherwise specified or contextually clear.

"Render" or "Rendered" or "Rendering"-how information (including text, graphics and / or electronic ink) is displayed, printed or The process of determining if the output is otherwise.

"Computer-Readable Medium"-any available medium that can be accessed by a user on a computer system. By way of example, “computer readable media” may include, but are not limited to, computer storage media and communication media. "Computer storage media" includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. "Computer storage medium" includes RAM, ROM, EEPROM, flash memory or other memory technology; CD-ROM, digital versatile disk (DVD) or other optical storage device; Magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices; Or any other medium that can be used to store desired information and can be accessed by a computer. A "communication medium" typically embodies other data or other transmission medium within a modulated data signal, such as computer readable instructions, data structures, program modules, or carrier waves, and includes any information delivery medium. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, communication media includes, but is not limited to, wired media such as a wired network or direct-wired connection and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of "computer readable media".

<Overview>

According to various embodiments of the present invention, the attributes of an element in a document (or other type of file) may include information regarding the spatial location of that element in the document. Thus, both the electronic ink stroke and the typed text can be described by the same spatial coordinate system and / or spatial relationship or other relationship with other elements within the document. Furthermore, related elements within a document can be identified simply by identifying the spatial area of the document in which these elements are included and / or by linking such elements.

Using this spatial relationship between various document elements, a software application can create and maintain a data structure that describes other relationships between document elements. For example, a software application can maintain a data structure, such as a data tree, that defines classes for various elements in a document. Thus, a node in a data structure may correspond to a picture or handwritten word comprising one or more ink strokes, and the data stored at that node may also indicate the location of that word on the page. Any other suitable data or conventional data typically stored in connection with the electronic ink word and / or ink stroke may also be stored in the word node.

This type of data structure can also associate document elements (such as individual ink strokes or typed text characters) with meaningful groups such as words, lines of words, sentences, paragraphs, and the like. Therefore, if a software application maintains a document tree structure describing a paragraph of handwritten electronic ink, the leaf nodes of that data structure may contain data about individual strokes of the electronic ink, and one or more strokes. Can be associated with each other in the data structure as word nodes corresponding to words in the paragraph (eg, words determined by a parser and / or recognizer). The tree structure can then associate the word nodes with the row nodes corresponding to the rows in the paragraph. Each line node may also be associated with a node corresponding to that paragraph. In addition, the software application may correspond to a node corresponding to an electronic ink stroke and / or some other group of electronic ink, corresponding to a non-ink document element such as an image, an ink drawing, typed characters, words, lines, paragraphs, or the like. You can maintain a tree or other data structure that associates with another node. Therefore, such data structures may be used to define relationships between associated electronic ink strokes, to distinguish electronic ink strokes that form handwritten text from electronic ink strokes that annotate non-ink document elements, and And / or may be used to link the electronic ink strokes to other document elements.

As will be discussed in detail below, these data structures can be used with ink analysis tools in accordance with various embodiments of the present invention to analyze electronic ink in a document. According to various embodiments of the present invention, a software application may first obtain an analysis of electronic ink in a document by creating a data structure for the document. The data structure describes if there is a relationship between already analyzed document elements, thus providing a context for any new electronic ink to be analyzed. This data structure or "analysis context object" also includes any new electronic ink that has not been analyzed. In other words, the analysis context object also includes an electronic ink that has not yet established a relationship with other document elements. In some examples of the invention, the software application creates the analysis context object by itself. However, as other examples of the invention, a software application uses an ink analysis tool or another tool to create an analysis context object.

After the software application provides the analysis context object to the ink analysis tool (or after the ink analysis tool generates the analysis context object), the ink analysis tool is informed about at least a portion of the analysis context object that contains the electronic ink that has not been analyzed. Search for it or make a copy of it. By retrieving or making a copy of the information about the desired portion of the analysis context object, the ink analysis tool creates a data structure that can be subsequently analyzed without changing the analysis context object maintained by the software application. That is, the copy is separate from the actual electronic document used in the software application, and thus can be referred to hereinafter as a "document independent" analysis context object.

Once the ink analysis tool has created a document independent analysis context object, the ink analysis tool provides the document independent analysis context object to one or more analysis processes. For example, if handwriting recognition is performed on an unanalyzed electronic ink in a document, the ink analysis tool may classify the ink into text and picture strokes (if necessary or required) and ink the unanalyzed electronic ink text strokes. A document independent analysis context object may be provided to the classification and / or layout analysis processes to group into associated groups based on the layout. While the classification step and / or layout analysis processes analyze the document independent analysis context object, the software application may continue normal operation. In particular, the software application may continue to receive new electronic ink input and / or any other data in the electronic document continued by the application program.

When an analysis process such as a parsing process has completed the analysis of the document independent analysis context object, the analysis results are returned to the ink analysis tool. In particular, the parsing process (among which, in particular, may include classification and layout analysis processes as described above) returns a modified version of the document independent analysis context object showing new relationships for the electronic ink that were not previously analyzed. However, since the software application is free to accept new electronic ink input and / or any other data for the document during the parsing operation described above, the current version of the analysis context object for the document (ie, the version maintained by the application) The document independent analysis context object originally provided with the ink analysis tool and the parsing results provided by the parsing process may be different.

Thus, as some examples of the present invention, the ink analysis tool can obtain a current version of an analysis context object from a software application, and can adjust parsing results using the current version of the analysis context object. During this adjustment process, the ink analysis tool will update the current version of the analysis context object to reflect the results of the parsing process. The ink analysis tool will then transfer the adjusted data from the current analysis context object to the handwriting recognition process for recognition. However, as other examples of the present invention, the ink analysis tool may omit the adjustment process, and instead pass the parsing results directly to the handwriting recognition process.

Once the parsing results have been adjusted using the current version of the analysis context object, the software application can return back to normal operation and thus continue to receive any other data and / or new electronic ink input about the document. . On the other hand, the recognition process analyzes the adjusted data (or its parsing results) from the current analysis context object. The recognition process returns recognition results to the ink analysis tool after completing the analysis of the adjusted data (or parsing results). Again, because the software application may have received any other data and / or new electronic ink input for the document during the operation of the recognition process, the ink analysis tool obtains the current version of the analysis context object from the software application. The ink analysis tool adjusts the results from the recognition process using the current version of the analysis context object to update the current version of the analysis context object with the adjustment results.

<Example Operating Environment>

1 shows a schematic diagram of a general-purpose digital computing environment that can be used to implement various embodiments of the present invention. In FIG. 1, computer 100 includes a system bus 130 that couples various system components including processing unit 110, system memory 120, and system memory 120 to processing unit 110. Include. System bus 130 may be a local bus that utilizes one of several types of bus architectures including a memory bus or a memory controller, a peripheral bus, and any of a variety of bus architectures. System memory 120 may include read only memory (ROM) 140 and random access memory (RAM) 150.

Basic input / output system 160 (BIOS) includes basic routines that help transfer information between elements in computer 100, such as stored in ROM 140 during startup. Computer 100 also includes a hard disk drive 170 for reading from and writing to a hard disk (not shown), and a magnet for reading from and writing to the removable magnetic disk 190. Disk drive 180 and optical disk drive 191 for reading from and writing to removable optical disk 192, such as a CD ROM or other optical media. The hard disk drive 170, the magnetic disk drive 180, and the optical disk drive 191 are connected to the system bus 130 by the hard disk drive interface 192, the magnetic disk drive interface 193, and the optical disk drive interface 194. Respectively). Such drives and their associated computer readable media provide non-volatile storage of computer readable instructions, data structures, program modules, and other data for the personal computer 100. Computer-readable media capable of storing computer-accessible data such as magnetic cassettes, flash memory cards, digital video discs, Bernoulli cartridges, random access memories (RAMs), read-only memories (ROMs), and the like. Those skilled in the art will appreciate that other types may be used in the exemplary operating environment.

Many program modules that may be stored on hard disk drive 170, magnetic disk 190, ROM 140, or RAM 150 include operating system 195, one or more applications 196, other program modules ( 197 and program data 198. A user may enter commands and information into the computer 100 through input devices such as keyboard 101 and pointing device 102 (such as a mouse). Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 110 via a serial port interface 106 connected to the system bus 130, but they are also connected to other ports such as parallel ports, game ports, or universal serial buses (USB). Can be connected by interfaces. In addition, these devices may be coupled directly to the system bus 130 via a suitable interface (not shown).

The monitor 107 or other type of display device may also be connected to the system bus 130 via an interface such as the video adapter 108. In addition to the monitor 107, a personal computer typically includes other peripheral output devices (not shown) such as speakers and printers. In one example, a pen digitizer 165 and accompanying pen or stylus 166 are provided to digitally capture freehand input. Although FIG. 1 illustrates a connection between pen digitizer 165 and serial port interface 106, in practice pen digitizer 165 may be coupled directly to processing unit 110, or as known in the art. May be coupled to the processing unit 110 in any suitable manner, such as via a parallel port or another interface and system bus 130. In addition, although the digitizer 165 is shown separated from the monitor 107 in FIG. 1, the usable input area of the digitizer 165 may be the same area as the display area of the monitor 107. Digitizer 165 may also be integrated into monitor 107 or exist as a separate device superimposed or added to monitor 107.

Computer 100 may operate in a network environment using logical connections to one or more remote computers, such as remote computer 109. The remote computer 109 may be a server, router, network PC, peer device or other common network node. In FIG. 1, only the memory storage device 111 is briefly illustrated, but typically a computer ( Many or all of the elements described above for 100). The logical connections shown in FIG. 1 include a local area network (LAN) 112 and a wide area network (WAN) 113. Such networking environments are commonplace in offices, large optical computer networks, intranets and the Internet using both wired and wireless connections.

When used in a LAN networking environment, the computer 100 is connected to the local area network 112 via a network interface or adapter 114. When used in a WAN networking environment, the personal computer 100 typically includes a modem 115 or other means for establishing a communication link over a wide area network 113, such as the Internet. The modem 115, which may be internal or external to the computer 100, may be connected to the system bus 130 via the serial port interface 106. In a networked environment, program modules depicted for the personal computer 100 or portions thereof may be stored in the remote memory storage device.

It is to be understood that the network connections shown are examples, and that other techniques for establishing a communication link between computers may be used. It is assumed that any of a variety of well known protocols exist, such as TCP / IP, Ethernet, FTP, HTTP, UDP, etc., and the system is configured to allow the user to retrieve web pages from a web-based server. Can be operated on. Any of a variety of conventional web browsers can be used to display and process data on web pages.

Although the environment of FIG. 1 illustrates an example environment, it should be understood that other computing environments may also be used. For example, one or more examples of the invention use fewer environments than all of the various embodiments shown in FIG. 1 and described above, which embodiments will appear in various combinations and subcombinations apparent to those skilled in the art.

2 illustrates a pen-input personal computer (PC) 201 that may be used in accordance with various embodiments of the present invention. Any or all of the features, subsystems, and functions of the system of FIG. 1 may be included in the computer of FIG. 2. The pen input personal computer system 201 includes a large display surface 202 on which a plurality of windows 203 are displayed, for example a digitizing flat panel display such as a liquid crystal display (LCD) screen. Using the stylus 204, a user can select a digitizing display area, display highlights, and write over it. Examples of suitable digitizing display panels include electromagnetic pen digitizers such as Mutoh Co. (now known as FinePoint Innovations Co.) or pen digitizers available from Wacom Technology Co. Other types of pen digitizers may also be used, such as optical digitizers and touch-sensitive digitizers. Pen-input computing system 201 uses stylus 204 to perform conventional application tasks such as manipulating data, entering text, creating, editing, and modifying spreadsheets, word processing programs, and the like. Interpret the gestures made.

The stylus 204 can be equipped with buttons or other features to improve its performance. In one example, the stylus 204 may be embodied as a "pencil" or "pen", with one end constituting the recording portion and the other end constituting the "eraser" portion moving on the display. Mark the portions of the electronic ink that should be erased. Other portions of input devices such as a mouse, trackball, keyboard, etc. can also be used. In addition, the user's own finger may be used to select or display portions of the image displayed on a contact or proximity-sensitive display. As a result, the term "user input device" as used herein is intended to have a broad definition and encompasses various variations of well known input devices.

In various examples, the system provides an ink platform as a set of Component Object Model (COM) services that an application can use to capture, manipulate, and store ink. The ink platform can also include markup languages, including languages such as Extensible Markup Language (XML). The system can also use DCOM as another implementation. In addition, implementations including the Win32 programming model and the Net programming model from Microsoft Corporation may be used. Such platforms are commercially available and known in the art.

In addition to the use of full-performance pen-type computing systems or "tablet PCs" (eg, convertible laptops or "slate" type tablet PCs), embodiments of the present invention may be hand-held or palm. Palm-top computing systems; Personal digital assistant (PDA); Pocket personal computers; Mobile and cellular telephones, pagers, and other communication devices; clock; Home appliances; Any other devices or systems, including a monitor or other display device, and / or present printed or graphical information to users and / or allow input using an electronic pen or stylus or be collected by another device Other types of pen-input computing systems, such as digitizers (e.g., conventional desktop computers capable of processing electronic ink collected by tablet PCs) and / or electronic ink. It can be used in other devices that accept data and / or accept an electronic pen or stylus input.

The invention will now be described with reference to the various embodiments of the invention and the remaining figures showing information for aiding the description. The specific drawings and information contained in the detailed description should not be inferred as limiting the invention.

<Spatial Document View Extraction>

As noted above, some embodiments of the present invention generally relate to systems and methods for providing richer and more versatile annotations using electronic ink in electronic documents. The following describes various embodiments and examples of the present invention in more detail.

A. Comprehensive Overview of the Invention

3 schematically illustrates the operation of a system and method in accordance with at least some examples of the invention. Specifically, as shown in FIG. 3, a user may interact with the electronic document 300 (document “A” in FIG. 3) by several methods, such as by adding electronic ink to the electronic document 300. Can be. In the illustrated example, the electronic document 300 includes electronic or typed text 302 ("Sample Text" in the illustrated example) and ink picture 304 (home in the illustrated example). Of course, the electronic document 300, such as electronic text, electronic ink (picture or text), images, graphs, tables, charts, other graphics or electronic information and / or combinations thereof, without departing from the invention. It can include any data or information. For the purpose of this example, the original electronic document 300 (also referred to herein as the "foundation document" or "foundation portion") includes electronic typed text 302 and electronic ink picture 304. When the user reviews document A 300 in this example, the user decides to include electronic ink annotation within document 300. Specifically, in the illustrated example, the annotation includes a circle 306 that surrounds a portion 308 of the electronic text 302.

In this example, after the user enters an annotation, the application will call parser 310 to request parsing of the newly entered electronic ink data (as described above, "parsing" is, for example, electronic ink). Categorizing into different ink types (e.g., pictures, text, flowcharts, music, etc.), ink layout analysis (e.g., identifying spatial and positional relationships between ink strokes and grouping them into appropriate groups) And other analytical processes). Specifically, the application program will send the inputted electronic ink data (in this example, representing the annotation 306) to the parser 310 as unclassified ink data. This is illustrated by input arrow 312 in FIG. 3.

Optionally, the application program may send some or all of the data associated with the base electronic document 300 to the parser 310 simultaneously or nearly simultaneously. Such data may include data regarding the spatial layout of information within document 300, such as, for example, the location of electronic text 302, picture 304, and the like. Additionally, the application can send data representing the hierarchical structure of the data in the document, which is described in more detail below. Alternatively, and as described in more detail below, the parser 310 relates to the electronic document 300, such as information about the electronic document 300 in the spatial area associated with the new, unclassified ink. Call back to the application requesting any data (in this example, data representing typed text and information spatially located near annotation 306). This data input to parser 310 is shown by input arrow 314 in FIG. 3. Of course, both the unclassified ink and the pre-analyzed data can be sent to the parser at the same time and / or made available to the parser without departing from the present invention.

The parser 310 takes the inputted electronic document data 314 and inputted unclassified ink data 312 and uses that information to classify the new electronic ink data into an ink type. Inks can be classified into various ink types without departing from the invention, and several examples of possible ink types are described in more detail below. In this example, the parser 310 according to this example of the present invention is based on input ink 306 and information 302 included in the base document 300, as annotations, or more specifically, specific typing. The input ink 306 can be determined and classified as an ink picture corresponding to a container annotation (eg, a circle) containing the text in question. Of course, highlighting annotations, "point to" annotations, "point from" annotations, anchor annotations (eg, not "point to" or "point prompt"), horizontal Various types of annotations are also possible, such as horizontal span, vertical span, and the like, and various examples will be described in more detail below.

Once parsed, the data associated with annotation 306 can be incorporated into a "document tree" hierarchical data structure (or any other suitable or desired data structure) associated with electronic document 300, and the data structure (or application). Instructions for changing the existing data structure maintained by &lt; RTI ID = 0.0 &gt;) may be sent back to the application, as indicated by arrow 316 in FIG. If desired, other data may also be returned from the parser 310 to the application without departing from the present invention. For example, as shown in FIG. 3, if the operation of the included parser 310 is called in one or more recognizer modules, components, and / or programs, the result of the recognition analysis process (such as a handwriting recognizer or other recognizer) May be returned to the application.

4 shows an illustration of an implementation 400 of various features used in some examples of the invention. As shown, an electronic document (such as document A in FIG. 3) is created, stored, and / or maintained by application 400a as a data structure or document model 402. This data structure 402 can be a hierarchical data structure or any other suitable or desired data structure without departing from the invention. Application 400a includes a "Analysis Context" object 404 that contains a mirror copy, an optional copy, or other relevant information about data structure 402 (eg, analysis). The context object 404 may be a "translation" layer, where queries for the analysis context structure 404 are translated into queries of a native document mode structure-in this way, Parsing techniques in accordance with embodiments of the present invention may be translated for use in a variety of different native applications). The operating system side or “platform” 400b of the implementation 400, which is the present example, includes an “ink analyzer” method 406 called by the application 400a. The application 400a calls the ink analyzer method 406 and passes this reference to the analysis context object 404. The ink analyzer method 406 will call the parser 408 (also part of the platform 400b in this example), which parses and analyzes the layout of the input data as outlined above. The operation of an exemplary implementation according to the present invention is described in more detail in the <analysis of ink> section below. Once the parser 408 has completed the analysis and processing, the platform 400b is parsed by the document model 402 and / or the analysis context object 404, as illustrated comprehensively by the arrow 410 in FIG. The data can be returned back to the application 400a to be able to rebuild the data structure based on the processing and results.
5 generates electronic document data in at least some examples of the invention, for example, as part of an “analysis context” object 404, as a document model 402, or by a parser 408; And / or one example of a data structure that can be used for storing as an output data structure. Specifically, FIG. 5 shows an exemplary electronic document 500. When the data structure is stored as an analysis context object 502 or subject to parsing, the data structure may include various document elements that are classified or arranged into different groups or "nodes". For example, as shown in FIG. 5, electronic document 500 may include three electronic text paragraphs 504, 506, and 508; One electronic ink drawing 510; And one electronic ink tin 512. An example analysis context data structure 502 for an electronic document 500 is shown as a list of top-level elements in FIG. 5. Specifically, node element 504 (a) corresponds to paragraph 504 in electronic document 500, node element 506 (a) corresponds to paragraph 506 in electronic document 500, and the node Element 508 (a) corresponds to paragraph 508 in electronic document 500, node element 510 (a) corresponds to paragraph 510 in electronic document 500, and node element 512 ( a)) corresponds to paragraph 512 in electronic document 500. As described in more detail below, additional nodes are parented to the top level nodes 504 (a), 506 (a), 508 (a), 510 (a), 512 (a) shown in FIG. It may be stored in a data structure (eg, in a hierarchical arrangement) as a node and / or child node.

delete

B. Context Nodes and Hierarchical Data Structures

6A-6I show various examples of data structures and context nodes that can be used for building, analyzing, and parsing electronic document data, such as an analysis context object, in accordance with some examples of the present invention. FIG. 6A shows an example data structure 600 that can be used to store unclassified electronic ink data (eg, ink newly input by a user that has not been previously parsed). This exemplary data structure 600 may be any suitable or desired group of groups (eg, corresponding to an entire electronic document in an application, corresponding to a specific page of a document, or used by an application). A root node 602 (which may correspond to data). The root node 602 may include any suitable or desired data, such as a page number, pointers to previous and next pages, location or size of margins, and the like. Each root node 602 may include any number of unclassified ink nodes 604 as child nodes, and each unclassified ink node 604 further includes any number of individual stroke nodes 606. (The last node in the hierarchy (in this example, the stroke node 606) may also be referred to herein as a "leaf node"). (In many figures, the letter “n” represents any number, including 0.) Various nodes of data structure 600 may include any desired or appropriate data corresponding to the node. For example, stroke node 606 may include data, which may include other data or digitizer points that identify an electronic ink stroke; Stroke position or orientation data; Stroke color data; Stroke pressure data; Stroke input timing data; Stroke ID data; And / or any other conventional or useful data used to store stroke information in systems and methods capable of receiving electronic ink. Similarly, unclassified ink node (s) 604 may include any desired or appropriate data associated with the ink or node (eg, bounding box size of all strokes contained within the node; bounding box location; input timing). Information representing, for example, geometric information about the ink (such as, for example, convex hull information); and / or data representing any conventional or useful other data relating to the unclassified ink strokes contained in the node. ) May be included. Alternatively, if desired, the stroke data may be stored as an attribute of another node (eg, word node, table node, etc.), rather than generally stored as a separate node of the document tree.

Once the unclassified ink is sent to the classifier, layout analyzer, recognizer and / or another parsing system, at least a portion of the ink may be associated with each other and / or classified into various different types, and the data structure associated with the ink may be It may be changed to correspond to various types and associations classified. 6B shows an example data structure 610 for electronic ink classified with ink text. Using parsing techniques, including conventional parsers known and used in the art, input electronic ink strokes (e.g., pen-down events and subsequent pen-up events) electronic ink data collected between pen-up events or in several other ways can be parsed and stored in a hierarchical manner, and the individual ink strokes involved can be grouped together and stored as ink words, linearly The placed ink words can be sorted together and stored as ink lines, the associated ink lines can be sorted together and stored as ink paragraphs, and the related paragraphs can be stored as electronic documents (e.g. as individual page parts, as a whole). Document, or in any suitable root group). As shown in FIG. 6B, an exemplary hierarchical data structure 610 corresponding to electronic ink data parsed and stored in this manner may include a root node 612, which may be a full electronic document. , Pages, or another group as used by an application. Each root node 612 may include any number of nodes 614 corresponding to similar groups or paragraphs of electronic ink strokes. Paragraph nodes 614 may include any number of individual line nodes 616, and individual line nodes 616 may include any number of individual ink word nodes 618, Individual ink word nodes 618 may also include any number of individual stroke nodes 620 corresponding to individual strokes of input electronic ink data. Optionally, as described above, the stroke data may be stored as "attributes" of the associated word node, as desired. The various nodes 612, 614, 616, 618, 620 may be any suitable data relating to the various individual strokes or set of strokes contained within the node, such as spatial orientation or positioning data, and / or as described above. The same other data can be stored.

Other data types and data structure arrangements corresponding to input electronic ink strokes are possible without departing from the invention. For example, classification, layout analysis, or recognizer techniques may determine that the input electronic ink forms an ink picture. 3 shows an example ink drawing 304 that includes several individual ink strokes. 6C and 6D show exemplary hierarchical data structures that can be used to group and store electronic ink data determined to relate to ink pictures. As shown in FIG. 6C, the data structure 630 includes a root node 632 (which may correspond to an entire electronic document, an electronic page, or some other data group, as described above). Each root node 632 can have any number of individual ink pictures (e.g., each individual picture has individual nodes that store data about, for example, picture width, picture height, picture position, bounding box size, etc.). Nodes 636 may be included, and each ink picture node 636 may include any number of individual ink stroke nodes 638 (corresponding to the individual ink strokes that make up the picture). . Optionally, the ink stroke data can be stored as the "attributes" of the corresponding ink picture node (or other node).

6D shows a replaceable data structure 630a for electronic documents that include one or more ink pictures. In this example, an electronic document or page or other group of data (denoted as root node 632a) is one or more groups of e-ink pictures with data stored and associated together in a hierarchical manner described in relation to FIG. 6C (denoted as node 634a). (E.g., an entire electronic document (denoted as root node 632a) may include several individual pages (each represented by group node 634a), each individual page having one or more individual ink drawings. (Each represented by ink picture node 636a), and each individual ink picture may include one or more individual ink strokes (each represented by stroke node 638a or indicated in the properties of the ink picture node). It should be understood that both groups and ink pictures can be children of the same root node or any group node. Also, if desired, one group node may include additional group nodes. Of course, other suitable data structures for storing electronic ink pictures may be used without departing from the present invention.

However, hierarchical data structures are not limited to use with input electronic ink data. Input typed text (eg, from a keyboard) can also be stored in a hierarchical data structure in association with various groups. 6E shows an example of such a data structure 640. This data structure 640 may include a root node 642 (eg, such as the root nodes described above). Each root node 642 may include any number of text paragraph nodes 644 (corresponding to paragraphs of text), and each paragraph node 644 may include any number of text line nodes 646. (Or alternatively text sentence nodes), each line node 646 may further include any number of individual text word nodes 648. If desired, text words can also be further broken down into individual text character nodes without departing from the invention, and individual character feature nodes (e.g., baseline, thin line, ascender and decidence of each character). Character feature node for sender features). The various nodes may include appropriate data about the text, such as paragraph, line or word spatial position, content, size, page margins; Margin size or location; Any suitable data about the number of pages and the like can be stored.

The various data structures described above with respect to FIGS. 6A-6E illustrate data structures that include exclusively electronic ink data only or exclusively include electronic typed text data. Parsing techniques can be used to analyze, combine, associate and group different data types within this single data structure without departing from the invention. In the example data structure shown in FIG. 6F, the electronic document or portion thereof (denoted as root node 652) is one or more individual paragraphs (denoted by paragraph node 654) and one or more individual lines (line node 656). Is indicated by). It is possible for both typed text (represented by text word node 658) and electronic ink text (represented by ink word node 660 and ink stroke node 662) to be included in the line. Each individual paragraph, line, and word node (both typed text and ink words) will contain appropriate independent data depending on the content of the paragraph, line, and word. Additionally or alternatively, the individual words may further include both ink letters and typed letters without departing from the present invention.

Other electronic data can also be included in the data structure of an electronic document (including hierarchical data structures) without departing from the invention. For example, as shown in data structure 664 of FIG. 6G, image data (such as digital photos, graphic information, etc.) may be included in the data structure for the electronic document. As shown in this example, data structure 664 includes an electronic document or portion thereof (denoted as root node 666), and the electronic document or portion thereof (denoted as root node 666) is one or more pieces of data. Individual groups (represented by group node 668). In this example, each data group is not only typed text (represented by paragraph node 672, line node 674, and text word node 676), as outlined above with respect to FIGS. 6E and 6F. Electronic image (represented by image node 670). Of course, data structures such as those shown in FIG. 6G also include electronic ink data (such as, for example, the data structures shown in FIGS. 6A-6D) or other desired data types without departing from the present invention. can do. 6H shows an example of another potential data structure 680 that can be used in accordance with at least some examples of the present invention. More specifically, data structure 680 of FIG. 6H corresponds to an electronic document or portion thereof (indicated as root node 682) that includes one or more lists (indicated by list node 684). The list in the electronic document may include any number of individual list items (represented by list item node 686), and each list item may contain a wide variety of different types of electronic information. For example, some list items may optionally include a list bullet, and the list bullet may consist of a typed bullet or an electronic ink bullet (including one or more corresponding ink strokes 690). Ink bullet node 688 is shown in the example of FIG. 6H). In addition, each list item may optionally include one or more electronic ink paragraphs (denoted by node chain 692), one or more ink pictures (denoted by node chain 694), and / or one or more typed paragraphs (node chains ( 696). Additionally (or alternatively), each list item may optionally include one or more images (represented by image node 698) and / or one or more data groups, such as the groups described above with respect to FIGS. 6D and 6G. (Represented by group node chain 700). The various nodes and / or node chains 692, 694, 696, 698, 700 generally refer to subsets and / or groups of such data structures as well as the example data structures shown in FIGS. 6A-6G. Since it may correspond, further description is abbreviate | omitted. If desired, in at least some examples, the individual list item node 686 itself may also include a list having a data structure as shown in FIG. 6H.

6I illustrates another exemplary data structure 710 that can be used in at least some examples of the present invention. This example data structure 710 is analyzed as a table and corresponds to electronic data that is stored together and associated together and grouped. In this illustrated example, each electronic document or portion thereof (denoted as root node 712) may include one or more tables (denoted as table node 714). In this example, the table may consist of one or more rows of information (represented by row node 716), and all rows may consist of one or more individual cells (represented by cell node 718). Individual cells may include one or more data structures, including, for example, any of the various different data structures described above. In the example shown, an individual table cell is any number of images (represented by image node 720), any number of typed text paragraphs (represented by node chain 722), any number of groups (Represented by node chain 724), any number of additional tables (represented by node chain 728), and any number of lists (represented by node chain 730). Is shown. Of course, without departing from the invention, any other suitable or desired data structures may be included in the table, including the various specific data structures described above and / or combinations and subsets of the data structures described above, for example. May be included.

Any suitable or desired data may be stored at various other nodes, including the various types of data described above. For use in processing data relating to annotations, it may be useful in at least some examples of the invention for various nodes to store data about the spatial position, orientation, or position of a node in the electronic document, optional This spatial information in turn relates to or links with other nodes in the electronic document data structure.

In addition, other types of parsers, classifiers and / or recognizers and data types may be used without departing from the invention. For example, recognizers may include electrical or electronic symbols (eg, resistors, voltage sources, capacitors, etc.) without departing from the invention; Music symbols; Mathematical symbols; Flowchart elements; It can be used to analyze, recognize, group, and / or associate pie chart elements and the like.

The data structures shown in FIGS. 6A-6I are merely examples of various data structures that can be used in accordance with embodiments of the present invention. Those skilled in the art will recognize that the specific data structures used may be variously modified within the scope of the present invention. For example, different nodes representing different layout groups may be used, additional node types may be added, or some of the described node types may be omitted without departing from the invention (eg, For example, within a table data structure, column nodes may be used instead of row nodes). In addition, the foregoing is optimized for use in language and textual representation in which characters on a page are read from left to right and top to bottom. Read and / or written in data structures designed for use in correspondence with other languages and character arrangements, e.g., right to left, top to bottom, top to bottom, or combinations thereof, without departing from the invention. Data structures to accommodate languages can be used. In addition, if necessary (especially for some languages, such as Asian languages), the strokes can be grouped properly into characters, and they are grouped back into lines, as described above, to "word" nodes. May not be used in the middle. Alternatively, the use of line nodes can be omitted and word nodes can be grouped together into paragraphs and / or other desired groups. Such variations and other modifications of the data structure to accommodate the characters of a particular language may be used without departing from the present invention.

C. Application of Embodiments of the Invention to Comments

As discussed above, embodiments of the present invention, including the use of context nodes and hierarchical data structures, can be used to provide "smart" electronic ink annotations to electronic documents. As noted above, annotating electronic documents is one of the preferred and advantageous methods of using pen-type computing systems. However, to be particularly useful and effective, annotations made with electronic ink on documents on pen-input computing systems, characters associated with various elements in a base document, for example, such as margin changes, display size changes, When changing fonts, adding information, deleting information, etc., the annotations must be flexible enough so that the position and / or other features of the annotations can be accurately changed based on changes to the annotated elements in the base document. . This may be accomplished, for example, by linking at least one context node associated with the annotation to one or more context nodes associated with the base document. In at least some examples, the data and / or characteristics associated with the base document and stored in relation to the base document, such as data relating to spatial attributes, position, and / or location of one or more document elements, It can be used to control the position, shape and / or other properties of the annotation.

7A and 7B show an example of annotation that can be applied to an electronic document through electronic ink. As shown in FIG. 7A, the electronic document 750 includes the sentence “Today the sky is green”. A user of a pen-type computing system annotated document 750 using electronic ink to underline the word “green” (underlined annotation is shown by reference numeral “752”). 7B shows an example hierarchical data structure 760 corresponding to such a compound electronic document 750 (the entire electronic document 750 is represented by a root node 762 within the data structure). Once fully parsed, data structure 760 will include context nodes showing the overall document structure. More specifically, electronic document 750 includes one paragraph (paragraph node 764), which paragraph includes a line (line node 766), and that line contains five words (words "Today"). information about typed text in electronic document 750 to indicate that it includes text word nodes 768a through 768e for ", ", " sky ", " is " and " green " Will be analyzed and associated together. Additionally, the parser will find the presence of data related to annotation 752 in electronic document 750 and classify this data as an electronic ink picture (ink picture node 770). Additionally, based on the location and location of the ink glyph stroke (s) relative to the typed text in the background document 750, the parser will recognize and classify the annotation as an "underlined" annotation and the underline node ( 772). Additionally, the parser will store all the stroke data corresponding to the actual ink stroke (s) of the underline annotation in one or more stroke leaf nodes 774 (optionally, the stroke data may be another suitable node or underline node 772). Can be stored as an attribute of).

Additionally, each time an individual word "green" appears in the electronic document 750, in order to keep the electronic ink underline stroke 752 with the individual word "green" (e.g., for some reason the word is moved for some reason) Case), as shown by arrow 776 in FIG. 7B, an underlined context node 772 (eg, a “source” context node in this link example) is a text word node for the word “green”. The data is stored in the data structure 760 of this example to indicate that it is linked to 768e, and the data is to indicate that the text word node 768e is linked to the underlined node 772, as indicated by arrow 778. Stored. This link “anchor” the underline node 772 to the text word node 768e (eg, a “coin” type link). Thus, for whatever reason, if the data stored in the text word context node 768e indicates that the position of the word "green" in the electronic document has changed, the links 776, 778, if possible, the text on the changed electronic document. Allows an application to detect that the rendering of underlined annotations must be moved to the spatial location and / or location of the word context node 768e. As another example, "bounding box" data, word width data, or other data stored in context node 768e may be for some reason (eg, changing font size, adding characters, deleting characters, changing characters). Indicative of a change in the size or position of a word, such as, for example, the application may, if possible, modify the size of the rendered underline (eg, lengthened or shortened) to match the text word represented by the context node 768e. To correspond to the associated new size. Where applicable, systems and methods in accordance with at least some examples of the present invention may split the stroke (s) of the annotation so that it appears on one or more lines, (eg, by adding words, hyphenation, etc.). If the underlying document changes, make sure that the annotated document elements appear on multiple lines. In addition, any suitable data indicative of any suitable link arrangement or link between nodes may be used without departing from the invention.

8A and 8B show another exemplary annotation. In this example, as shown in FIG. 8A, the electronic document 800 includes the sentence "Today the sky is green", and the user of the pen-input computing system lines up the word "green" and deletes it (strike out). The document 800 was annotated using electronic ink for marking (strike-out marking is shown by reference numeral 802). FIG. 8B shows an exemplary hierarchical data structure 810 that is fully parsed and corresponds to the composite electronic document 800 (since many specific context nodes of FIG. 8B correspond to the context nodes in FIG. 7B, FIG. The same reference numerals are used as in 7b, and duplicate descriptions are omitted). In this case, the parser will know the presence of data in the electronic document 800 associated with the annotation 802 and classify the data as an electronic ink picture (ink picture node 812). Additionally, based on the location and location of the ink glyph stroke (s) relative to the content of the background typed text in document 800 (eg, the strikeout marking crosses the text word horizontally). The parser will recognize and classify the annotation as a "strike-out" annotation and create a strikeout node 814. In addition, the parser stores all of the stroke data corresponding to the actual ink stroke (s) of the strikeout annotation (or, alternatively, as an attribute of strikeout node 814 or another suitable node). 816).

As in the case of the underline comment described in connection with FIGS. 7A and 7B, when the individual word "green" moves for some reason, the strikeout annotation may be retained with the individual word "green". This is illustrated, for example, by the strikeout context node 814 (in this example, the "source" context node), as shown by the arrow 818 in FIG. 8B, the text word node 768e for the word "green" ( By storing data in the data structure 810 to indicate that it is linked to a "destination" context node in this example, it is also indicated that the text word node 768e is linked to the strikeout node 814 as indicated by arrow 820. By storing the data in a data structure for presentation. The link type in this example is a "horizontally spanning" link type that anchors strikeout node 814 to text word node 768e. Thus, if the data stored in text word context node 768e indicating the location and / or size of the word "green" (or other word (s) associated with node 768e) has changed in document 800 for some reason, the link The 818, 820 may adjust the rendering of the strikeout if possible and / or (if possible) the size of the words stored in the context node 768e in the modified electronic document, and / or the spatial location, the size of the place of existence. It will allow the application to detect that it needs to be adjusted. Again, any suitable link arrangement or link between the nodes may be used without departing from the present invention.

9A and 9B illustrate another exemplary type of annotation, further illustrating that the annotations are associated or linked with one or more text words. In the example shown in FIGS. 9A and 9B, the electronic document 900 includes the sentence "Today the sky is green", but in this case, a user of the pen-input computing system circles the words "is green". An annotation was marked on document 900 using electronic ink for marking (an annotation of the circle or “container” type is shown at 902 in FIG. 9A). FIG. 9B shows an exemplary hierarchical data structure 910 that is fully parsed and corresponds to the compound electronic document 900 (many specific context nodes of FIG. 9B correspond to the context nodes in FIG. 7B, and therefore in FIG. 9B). The same reference numerals as those present in 7b are used, and duplicate description thereof is omitted). In this case, the parser will recognize the presence of data in the electronic document 900 associated with the annotation 902 and classify this data as an electronic ink picture (ink picture node 912). Additionally, based on the location and location of the ink glyph stroke (s) relative to the content of the background typed text in document 900, the parser recognizes and classifies the annotation as a "container" annotation and data. Will create container node 914 in structure 910. Additionally, the parser stores in one or more stroke leaf nodes 916 all stroke data corresponding to the actual ink stroke (s) of the container tin (or, alternatively, as an attribute of an ink container node or another suitable node). will be.

As with the underline and strikeout annotations described in connection with FIGS. 7A-8B, container annotations can be retained with the word "is green" even if the words "is green" move for some reason. This means, for example, that the ink container context node 914 (in this example a "source" context node) is the text word nodes for the words "is green" (as shown by arrow 918a in FIG. 9B). 768d, 768e (in this example, " destination " context nodes) may be achieved by storing data in data structure 910 to indicate linking to each. Additionally, as indicated by arrows 920a and 920b in FIG. 9B, data may be stored in data structure 910 to indicate that text word nodes 768d and 768e are linked to container node 914. This link is a "containment" type of link. Thus, the data stored in the text word context nodes 768d, 768e may cause the electronic document 900 for some reason due to the location and / or size of the words “is green” (or other word (s) associated with these context nodes). Link 918a, 918b, 920a, 920b, if possible, is located at the size, spatial location, and / or place of existence of the words stored in context nodes 768d, 768e in the changed electronic document. It will allow the application to detect that its size and / or container annotations need to be adjusted. Additionally, if the user adds words between the word "is" and the word "green", or if the words or letters are removed or changed in any way, the application can determine the size and size of the container annotation to accommodate these changes. You can modify the location. Within the scope of the present invention, other link arrangements or data representing a link may be used.

10A and 10B show another type of comment that is commonly used, especially the comment of the "margin comment" type. In this example, the user is directed to an electronic ink margin comment 1002 that describes "it's not green!" For an electronic document 1000 that includes the sentence "Today the sky is green", as shown in FIG. 10A. Commented out. Obviously, as shown in FIG. 10B, in this example, the text is present in two lines in the electronic document 1000 (unlike that in one line in FIGS. 7A, 8A, 9A). The data structure 1010 associated with the text has changed. Thus, the data structure 1010 has two line nodes 1012 and 1014, and the first line node 1012 is associated with the text word nodes 768a and 768b (associated with the words "Today" and "the"). And second line node 1014 includes text word nodes 768c, 768d, 768e (associated with the words “sky”, “is”, and “green”).

Once the composite electronic document 1000 is fully parsed, the parser will recognize that the annotation 1002 contains electronic ink text, and this annotation is due (eg, due to the position of the margin of the document 1000). They will be categorized as comments of the "Margin Comments" type. Thus, the parser will create an appropriate context node 1016 for the margin comment. This margin comment 1002 contains only electronic ink text, so that the parser is a paragraph node 1018, two line nodes 1020, 1022, and appropriate ink word nodes (1024, 1026) associated with the hierarchical structure of the electronic ink. 1028) and stroke nodes 1030, 1032, 1034 (as described above, optionally and alternatively, ink stroke data may be stored as attributes of the ink word node (s), and / or Line nodes may be omitted in data structure 1010).

Another link relationship is shown in the example of FIGS. 10A and 10B. Specifically, in this example, based on the position and orientation of the margin comment 1002 with respect to the page margin and the content of the background document, the parser, as shown by the arrow 1036 in FIG. 10B, indicates the margin comment 1002. The data structure 1010 to indicate that paragraph node 1018 (in this example, a "source" context node) associated with is linked to paragraph node 764 (in this example, "destination" context node) associated with typed text. ). Additionally, the parser displays the data structure to indicate that paragraph node 764 associated with the typed text is linked by paragraph node 1018 of margin comment 1002, as shown by arrow 1038. Will save to 1010. This is a "fixed" type of link placement in which an ink paragraph is anchored to a text paragraph. In this manner, every time a typed paragraph moves within the electronic document 1000, optionally, additional lines and / or words are added to the paragraph, and / or lines and / or words are removed from the paragraph. , And / or other changes are made within the paragraph or in the electronic document 1000, it is necessary to move the margin comment 1002 so that the margin comment 1002 remains in the margin near the linked typed paragraph. The detection by the application is allowed by links 1036 and 1038. Of course, other link arrangements or data associated with the links can be stored without departing from the invention. For example, the margin comment node 1016 may be linked to one or more individual text word nodes of the typed paragraph, such as the first text word node 768a.

11A and 11B show more or less complex illustrations of annotations recognizable by a method and system in accordance with at least some examples of the present invention. In the example shown, the electronic document 1100 includes the phrase "Today the sky is green", but in this case, the user has annotated the electronic document 1100 using a combination of different annotation types. Specifically, the user may have a container type annotation 1102 surrounding the word "green", a margin comment type annotation 1104 including the electronic ink word "Blue!", And a margin comment annotation 1104 from the container type annotation 1102. A connector type tin 1106 (arrow in this example) is shown.

Due to the relative position and spatial orientation of the various document types relative to the base document, typed document text, and the margin of the document, the parser system according to at least some examples of the present invention, as outlined above, , Their content, their relationship to each other, and their electronic document typed text. An example hierarchical data structure 1110 for the electronic document 1100 generated by the parser is shown in FIG. 11B. Since the hierarchical structure for the typed text is the same as that shown in FIG. 7B, the same reference numerals are used in FIG. 11B, and a detailed description thereof will be omitted. Also, as shown in FIG. 11B, the node chain 1112 for ink container type tin 1102 is similar to that shown in FIG. 9B, and the node chain 1114 for margin comment type tin 1104 is shown in FIG. Similar to that shown in 10b. Therefore, detailed description of these node chains is omitted.

In order to maintain the appropriate portion of the base text and the spatial position of the annotations together, several links between annotation nodes and several links with appropriate text nodes are provided by the parser and stored in the data structure 1110. In the illustrated example, as shown by arrow 1118 in FIG. 11B, the ink container node 1116 (where the “source” context node) is linked to the text word node 768e of the word it contains. . Similarly, as indicated by arrow 1120 in FIG. 11B, data is stored in data structure 1110 to indicate that text word node 768e is linked to ink container node 1116. This is an "container" type link array.

As mentioned above, the parser according to this example of the present invention is characterized in that the electronic document 1100 includes an ink picture 1106 corresponding to an ink connector (i.e., in this example, between container annotations 1102 and margin comment annotations 1104). Arrow). This connector annotation 1106 is stored in the data structure 1110 as an ink picture (indicated by an ink picture node 1122 in this example), and the ink picture is a node representing an annotation type (in this example, an ink connector node ( 1124), and the node representing the annotation type may include the stroke or strokes making up the connector (e.g., with attributes associated with the stroke node (s) 1126, or other suitable node or ink connector node 1124). It further includes data indicating. In addition, information is sent to the data structure 1110 to indicate that the ink connector node 1124 (the "destination" node in this example) is pointed from the ink connector node 1116 (the "source" node in this example). Stored. This link (link of type “points from”) is indicated by arrow 1128 in FIG. 11B. In addition, the data structure 1110 displays data to indicate that the ink connector node 1124 (the "source" context node in this example) is linked to the ink word node 1132 (the "destination" context node in this example). It includes more. This link ("point to" type of link) is indicated by arrow 1134 in FIG. 11B. Alternatively, ink connector node 1124 may be linked to any of the ink line node, ink shorting node, or margin comment node of node chain 1114 without departing from the invention. Using the specific links described above, the data structure 1110 can be used to ensure that the ink connector 1106 is pointed from the circle picture 1102 and pointed to the ink word, as shown in FIG. 11A. Indicates. These various links allow the application to detect the relationship between the base document and the various annotation elements. If possible, the application can then move the annotation elements to appropriately correspond to the movement in the base document. Any suitable link arrangement or data associated with a link may be used without departing from the invention.

12A and 12B show another exemplary annotation feature that is generally used. As shown in FIG. 12A, the electronic document 1200 in this example includes the sentence “Today the sky is green”, but in this case, a user of the pen-input computing system is placed near a first word in the sentence. Annotated the document 1200 using electronic ink to mark a star or an asterisk (annotation is shown by reference number 1202). FIG. 12B shows an example hierarchical data structure 1210 corresponding to compound electronic document 1200, once fully parsed (as many context nodes in FIG. 12B correspond to context nodes in FIG. 7B, FIG. In FIG. 12B, the same reference numerals are used as in FIG. 7B, and redundant description thereof will be omitted). In this case, the parser will recognize the presence of electronic ink data for the annotation and will classify this data as an ink picture (ink picture node 1212). Additionally, based on the ink stroke shape as well as the location and point of the ink stroke (s) of the annotation 1202 relative to the content of the base document 1200 (eg, parallel to the lines of typed text). , The parser will recognize the comment as a "star", "bullet" or fixed type of comment, and will create an asterisk node 1214. Additionally, the parser will store all stroke data corresponding to the actual ink stroke (s) of the asterisk as an attribute of one or more stroke leaf nodes 1216 or (some other nodes or) its asterisk nodes.

In this example, if the typed text moves for some reason, an asterisk type annotation 1202 may be maintained with that line every time an entire line of typed text appears in the electronic document 1200. This is, for example, an asterisk context node 1214 ("source" context node in this example) as shown by arrow 1218 in FIG. 12B, and text line node 766 ("in destination" context in this example). Storing the data in the data structure 1210 to indicate that it is linked to a node), and also to indicate that the text line node 766 is linked by an asterisk node 1214, as indicated by arrow 1220. Thus, if the data stored in text line context node 766 indicates that the point or location of the line on electronic document 1200 has changed for some reason, the context in the changed electronic document, if possible It is desirable for the application to detect that the rendering of the asterisk should be coordinated with the point and / or spatial location corresponding to the rendering of the data associated with node 766. Links 1218 and 1220. Alternatively, if necessary, the parser may include a star node 1214 (or other appropriate node associated with a star annotation) and a paragraph node 764 without departing from the present invention. ) Or one of the individual word nodes (e.g., word node 768a) .Also, as needed, ink picture node 1212 may be a source node (or bone) fixed to line node 766. In another example, other suitable nodes corresponding to the typed text may be used within the scope of the present invention without departing from the present invention.

The “flowchart” type of annotation linked in FIGS. 13A and 13B is shown. In this example, the electronic document 1300 includes a flowchart, which may optionally be included as part of an annotation within the large electronic document as a whole. As shown in FIG. 13A, the tin or flowchart is surrounded by or surrounded by a first ink entry 1302 (letter “A” in this example) and a second container tin 1308 surrounded or surrounded by a first container. A second ink entry 1306 is included. Additionally, connector tin 1310 (arrow in this example) extends from first container tin 1304 to second container tin 1308. However, it should be understood that relationships between document elements may be represented in alternative ways other than links. For example, such relationships may be represented in the structure of the data tree itself, for example through the use of container nodes containing associated document elements.

FIG. 13B shows an example data structure 1320 that may be associated with flowchart annotations of the electronic document 1300 of FIG. 13A once fully parsed. In this example data structure 1320, a root node 1322, which may correspond to all or a portion of the electronic document 1300, is provided to nodes in a flowchart type annotation as parent node. The first context node for the particular annotation structures is an ink picture node 1324 corresponding to container 1304. Ink picture node 1324 is provided with an ink container node 1326 as a child node, and the specific stroke (s) associated with ink container annotations 1304 are stored in leaf node (s) 1328 or an ink container node ( 1326 (or other node) are stored as attributes.

As described above with respect to FIG. 13A, the annotation further includes electronic ink text corresponding to the letter “A” 1302. In common with the other electronic ink text described above, the electronic ink associated with the ink text is stored as a paragraph node 1330 in the hierarchical data structure 1320, the paragraph node 1330 including a line node 1332, and a line Node 1332 includes ink word node 1334, which selects one or more leaf nodes 1336 corresponding to an attribute comprising the individual strokes in the ink text annotation or the stroke data. Include as. The information stored in the data structure 1320 is shown in FIG. 13B by the ink container node 1326 ("source" node) of the container annotation 1304 being a paragraph node of the letter "A". 1330 ("Destination" context node). Alternatively, if desired, ink word node 1334 or line node 1332 can serve as a destination node. Additionally, the information stored in data structure 1320 indicates that shorting node 1330 is linked by ink container node 1326, as indicated by arrow 1340 in FIG. 13B. In this example, the data structure for the ink container 1308 and the letter "B" 1306 of the annotation and the data structure for the ink container 1304 and the letter "A" 1302 includes the same general link structure. Because they share the same general structure, the various nodes of the data structure 1320 for the letter "B" 1306 and the associated container 1308 share the same reference numerals as the letter "A" and the associated container, but the letter "b &Quot; follows reference numerals for the ink container 1308 and the ink word 1306 in FIG. 13B. Of course, various hierarchical arrangements and link structures or link data may be different from those described above, without departing from the invention.

The annotation further includes a connector annotation 1310 represented by an ink picture node 1342 in the data structure 1320. As described above with respect to FIG. 11B, the data structure for the ink picture representing the ink connector annotation may further include an ink connector node 1344, wherein the ink connector structure 1344 may include one or more attributes or leaf nodes. (S) 1346 may be further included, and leaf node (s) 1346 include data associated with the particular stroke (s) that make up connector 1310. To complete the annotation data structure of this example, data associated with ink connector annotation 1310 (source node) is linked to data associated with two container annotations 1304 and 1308 (destination nodes). Specifically, as shown in the example of FIG. 13B, to indicate that the ink container node 1326 is linked to the ink connector node 1344 as a "point prompt" annotation type (indicated by arrow 1348), the information is displayed. Stored in structure 1320. Additionally, information is stored in data structure 1320 to indicate that the ink connector node 1344 is linked to ink container node 1326b as a "point to" annotation type (indicated by arrow 1352). In this manner, the data stored for the links indicates that the connector 1310 points from the circle container 1304 to the circle container 1308, as shown in FIG. 13A. Thus, by the presence of several links, it is possible to keep the annotations intact, even if the electronic document 1300 changes in some way, as other portions of the annotation are followed when the portion of the annotation in the electronic document 1300 moves. Do. Of course, this overall structure may be linked to some of the underlying document elements described above without departing from the invention.

Similarly, a comment may include two enclosures connected by a connector to a written comment such as, for example, "switch these!". If a part of a comment is moved, such as when words are inserted between two enclosures, that part is required to be moved independently of the other parts of the comment. Links between context nodes representing electronic ink annotations and nodes representing non-ink content can be used by a software application to reposition pieces of annotation and tie them back together when the positioning operation is complete. .

14A and 14B show another example of an electronic document 1400 having a flowchart type annotation or ink structure. In this example, an electronic ink annotation for the letter "A" 1402, by an electronic ink connector 1408 between the letters "A" and "B" and an electronic ink connector 1410 between the letters "A" and "C". It is linked to the electronic ink annotation for this letter "B" 1404 and letter "C" 1406, respectively.

In FIG. 14B, an example data structure 1420 is shown for an electronic document 1400 after parsing. Parent root node 1422 may store data relating to the entire electronic document or portions thereof. The data corresponding to the electronic ink words "A", "B" and "C" are the same paragraph nodes 1424a, 1424b, 1424c, line nodes 1426a, 1426b, 1426c as those used in some of the figures described above. Ink word nodes 1428a, 1428b, 1428c, and stroke node (s) 1430a, 1430b, 1430c. The repetition of the description of these nodes is omitted. Likewise, the ink connectors have the ink picture nodes 1432a and 1432b, the ink container nodes 1434a and 1434b and the stroke node (s) 1436a and 1436b used in FIGS. 11B and 13B, so that the repeated substrate Omit. Of course, variations of specific data structures may be used including the possible variations described above without departing from the invention.

To complete this example annotation data structure, the data associated with the ink connector annotation 1408 is linked to the data associated with the two ink word annotations 1402 and 1404 that are connected, and the data associated with the ink connector annotation 1410 Linked to data associated with the two ink word annotations 1402 and 1406 that are connecting. Specifically, as shown in the example of FIG. 14B, the information indicates that the ink picture node 1432a (“source” node) points to “point to” ink word nodes 1428a, 1428b. Stored in structure 1420 (indicated by arrows 1438 and 1440, respectively). Similarly, information is stored in data structure 1420 to indicate that ink picture node 1432b (“source” node) points to ink word nodes 1428a, 1428c (arrow 1442, 1444). Ink word nodes 1428a, 1428b, 1428c function as "destination" nodes. Because of the various links, if a portion of the annotation of the electronic document 1400 is moved, other portions of the annotation may follow, the annotation may remain intact or linked no matter how the electronic document 1400 is changed in any way. have. In addition, annotations may be linked with one or more other elements in an electronic document, such as typed text, pictures, images, and the like, without departing from the invention.

15A-15C show examples of electronic document 1500 and example data structures including electronic ink annotation in the form of a table having a plurality of rows and columns. Specifically, as shown in the example of FIG. 15A, the electronic document 1500 includes a table 1502 (consisting of four cells in total) with two rows and two columns, each cell of the table being an electronic ink. Consists of boundaries and contains electronic ink text input.

15B shows an exemplary data structure 1510 after completion of parsing for the electronic ink table 1502 shown in FIG. 15A. In this example, data structure 1510 includes a root node 1512 that may correspond to all or part of electronic document 1500. Below the root node 1512, the relevant data about the table is stored in the table node 1514. This may include, for example, data indicative of table size, table position, number of rows, number of columns, bounding box size, and the like. Below the table node 1514, separate row nodes are provided for each row of the table (in this example, two row nodes 1516, 1518 are provided, as shown in FIG. 15B). The row node of each of these exemplary data structures further includes one or more cell nodes (in the example shown, row node 1516 includes two cell nodes 1520 and 1522 and row node 1518 is Two cell nodes 1524, 1526). Since each cell in this example contains electronic ink text data, the remainder of the hierarchical data structure 1510 in this example includes the familiar paragraph node, line node, ink word node and stroke node (s) described above. Have Of course, the table cell may, for example, be in addition to typed text data and / or electronic ink data including the various data types described above with respect to FIG. 6I without departing from the invention, and / or Instead, it can contain one or more other data types. As another alternative, instead of row nodes 1516 and 1518, the data structure may include column nodes, which may include, for example, cell nodes, shorting nodes, and the like, as shown in FIG. 15B. can do. The ink strokes that make up the actual table may be, for example, as part of table node 1514 (eg, as attributes associated with that node), and / or (with associated strokes stored beneath it). It may be stored as an ink picture node depending on node 1514 and / or at any other suitable location within data structure 1510.

In FIG. 15C, an alternative exemplary data structure 1530 for an electronic document 1500 that includes a table 1502 is shown. In this example, root node 1512, table node 1514, row nodes 1516, 1518 and cell nodes 1520, 1522, 1524, 1526 are the same as shown in FIG. 15B. In FIG. 15B, the electronic ink text data is included as child nodes under the various cell nodes 1520, 1522, 1524, and 1526, whereas in FIG. 15C, paragraph nodes 1528a, 1528b, 1528c, and 1528d of the electronic ink text data are included. (Or any other suitable or desired data structure) is linked to cell nodes 1520, 1522, 1524, 1526, respectively, as shown by arrows 1530a, 1530b, 1530c, 1530d in FIG. 15C. In this example, the cell nodes may serve as container type "source" nodes for the included ink words, and the ink paragraph, line or word nodes serve as "destination" context nodes. By means of several links, moving some of the table annotations in the electronic document 1500 may lead to other parts of the annotations, so that the table annotations remain intact or linked even if the electronic document 1500 is changed in some way. There will be. The table can also be linked to some other nodes that exist based on the base electronic document. Again, as an alternative, the row nodes 1516 and 1518 may be replaced with column nodes comprising several cell nodes, without departing from the invention. In addition, the ink strokes that make up the actual table may be within any location, for example, as part of table node 1514 (eg, as attributes associated with that node) and / or without departing from the invention. Or as an ink picture node dependent on table node 1514 (with associated strokes stored below), and / or stored in any other suitable location within data structure 1510.

Of course, the various figures associated with the various exemplary annotation types described above are merely examples of information that may be included in the annotation, the use of the annotation, and possible annotation types. Those skilled in the art will recognize that many variations and modifications can be made without departing from the invention. For example, annotations may be substituted for many different data types (such as electronic ink text, pictures, images, typed text and images, etc.), in many different combinations and exchanges, without departing from the invention. It may include.

In addition, various linking schemes are described with respect to the various data structures described above. These link schemes or techniques are merely examples of how to allow several data nodes to be linked together. Any associated link arrangement or data associated with the link may be stored without departing from the invention. For example, it may be possible to represent each node as an individual identifier (such as a “globally unique identifier”, ie, a GUID), and to indicate its linking using GUID pairs (or more). As another alternative, instead of the dual link arrangement described in connection with some examples above, a single link may be used for the link nodes. As another possible alternative, the nodes can be linked to themselves if needed or desired (eg, self-linking where the source and destination nodes are in the same node). Those skilled in the art will recognize that any suitable manner of associating or linking various data sets with each other may be used without departing from the invention.

<Analysis of Ink>

DETAILED DESCRIPTION Various exemplary techniques for analyzing electronic ink in an electronic document in accordance with examples of the present invention are now described. In particular, FIGS. 16A-16E illustrate a flow chart showing steps of analyzing a document in accordance with various embodiments of the present invention. 17-26 then show the relationships between the different components used during the analysis process.

Referring to FIG. 17, this figure shows a software application 1701. Software application 1701 maintains document 1703, which may include both electronic ink data 1705 and non-ink data 1707, such as typed characters or images. As described in detail above, the attributes of the electronic ink data 1705 and the non-ink data 1707 may be represented by a hierarchical data structure such as a tree. To begin the analysis of the electronic ink, at step 1601, the software application 1701 creates a data structure identified as the analysis context object 1709 in FIG. 17.

18 shows an example tree 1801 of the type that may be included in the analysis context object 1709. Tree 1801 includes root node 1803 and shorting node 1805. Paragraph node 1805 corresponds to, for example, a paragraph of electronic ink text that may have been identified earlier in the preceding analysis process. Paragraph node 1805 is connected to two line nodes 1807 and 1809 representing two lines in a paragraph of electronic ink text. Line node 1807 is in turn associated with two word nodes 1811, 1813. Each word node 1811, 1813 corresponds to a word in a line of electronic ink text represented by the line node 1807. Word node 1811 includes ink stroke data 1815 and ink stroke data 1817. Therefore, the stroke data 1815 and 1817 correspond to the text electronic ink strokes that make up the word represented by the word node 1811. Similarly, word node 1813 includes ink stroke data 1819 and ink stroke data 1821 corresponding to text electronic ink strokes making up the word indicated by word node 1813. Tree 1801 also includes stroke data 1823-1827 associated only with root node 1803. This ink data 1823-1827 indicate new ink strokes that are unclassified or associated with another ink stroke or other document element.

Therefore, instead of storing the strokes in separate stroke nodes, the various nodes may have an associated "strokes" attribute that stores data corresponding to the strokes associated with the node. For example, within a list item (a) an unclassified context node may include one or more "stroke" attributes with one or more strokes requiring analysis, and (b) the ink word node constitutes an ink word. May include one or more "stroke" attributes that include one or more strokes; (c) the picture node may include one or more "strokes" attributes that include one or more strokes that make up the picture, and (d) one or more "strokes" that the bullet node includes one or more strokes that make up the bullet. It can contain attributes. Of course, as alternative implementations of the present invention, in association with the word node or picture node and the ink strokes, the individual stroke nodes can be used to represent, for example, the individual strokes of the former in the data tree.

The tree 1801 shown in FIG. 18 includes ink strokes already organized into words, lines, and paragraphs, while the tree 1801 is a new ink stroke that is not associated or classified with another ink stroke or document element. It should be understood that it can include only. For example, when a user first enters electronic ink into document 1703, this initial ink stroke would not have been analyzed. It should also be understood that tree 1801 is only a representative example and is very simplified to facilitate understanding of various embodiments of the present invention. For example, typically a line node will be associated with a plurality of word nodes, each word node may contain stroke data for several ink strokes. In addition, the tree 1801 includes only ink-related nodes while the analysis context object 1709 represents nodes that represent non-ink document elements, such as images and typed text, as described in detail above. It should be noted that they may include.

Returning to FIG. 17, as some examples of the present invention, the software application 1701 will create and maintain its own analysis context object 1709. For these software applications 1701, the software application 1701 may simply provide a reference to the analysis object 1709. However, for these software applications 1701, the software application 1701 must include the mechanisms needed to create, populate, and maintain an analysis object. However, some software developers will not want to raise the problem of providing the application 1701 to these mechanisms.

Thus, however, as other examples of the present invention, the software application 1701 may instantiate another object to create the analysis context object 1709. For example, software application 1701 may use an ink analysis tool or other object to create and / or maintain analysis context object 1709. For example, software application 1701 may provide or identify unanalyzed ink data to an ink analysis tool. Depending on the instructions from the software application 1701, the ink analysis tool may only scan the specific document (or regions) of the document containing the unanalyzed ink identified by the software application 1701, or the entire document 1703. An representing analysis context object 1709 may be generated. In some software applications 1701, the software application 1701 may then maintain and update the analysis context object 1709 on its own or using a service provided by an ink analysis tool or another object. can do. In still other software applications 1701, the software application 1701 may not hold the analysis document object 1709, but instead create a new analysis document object 1709 when needed by an ink analysis tool or other object. You can do that.

Typically, analysis context object 1709 will contain information about document elements for the entire document 1703. However, in some examples of the invention, it should be appreciated that the analysis context object 1709 may include information about document elements only in part of the document 1703. In other words, analysis context object 1709 may represent only document elements within the spatial area of the document that contain new or “dirty” ink or other data. If all document elements in the area within the document containing the electronic ink have already been analyzed, these already analyzed document elements may not be included in the analysis context object 1709. As another alternative, the application may maintain a separate analysis context object 1709 for each page or other subset of data related to the entire electronic document 1703.

Once the software application 1701 has created the analysis context object 1709, at step 1603, the ink analysis tool 1901 copies at least a portion of the analysis context object 1709 as shown in FIG. 19. More specifically, if the analysis context object 1709 is not limited to areas containing new ink and / or other data as described above, then the software application 1701 may need new ink and / or other data to be analyzed. Will specify areas of the document 1703 that include. The software application 1701 then invokes the ink analysis tool 1901 to copy the portion of the analysis context object 1709 that corresponds to the designated area of the document 1703. (Of course, if the analysis context object 1709 is already limited to describing regions containing new ink, the ink analysis tool 1901 may copy the entire analysis context object 1709.)

 The application 1701 will know which ink is the ink that has not been analyzed previously, but the portion that affects the new ink is which portion (s) of the electronic document 1703 (e.g., background data in the electronic document, Ink (analyzed, etc.) analyzed above, and therefore (which part (s) are portions to be provided to the ink analysis tool 1901). Therefore, in at least some examples of the invention, the application 1701 will make large sections available to the ink analysis tool 1901 via the analysis context object 1709 of the electronic document 1703. The application 1701 leaves the ink analysis tool 1901 to determine the information actually required by the application 1701 via the analysis context object 1709.

In response to a call to analyze new data, ink analysis tool 1901 may, if necessary, analyze contextual object 1709, necessary to analyze the new ink and / or other data within the area designated by software application 1701. Various callbacks are made to the analysis context object 1709 to obtain information from it. For example, the ink analysis tool 1901 may query about the analysis context object 1709 for new ink and / or other data, as well as information about other data and / or ink within a specified area that has already been analyzed, Or query the analysis context object 1709 for information about non-ink document elements within its designated area. A "designated area" may, in at least some examples of the invention, correspond to data of a spatial area in and / or in the vicinity of the area containing the new ink or other data to be analyzed.

Both pre-analyzed ink and non-ink document elements, especially those located near the newly input data to be analyzed, can provide a context for improving the analysis of unanalyzed ink or other data. Since the ink analysis tool 1901 obtains the desired information from the analysis context object 1709, the ink analysis tool 1901 generates a document independent analysis context object 1903 with the obtained information. Thus, document independent analysis context object 1903 includes a subset of the information contained in analysis context object 1709, but this is independent of document 1703. By using this "call back" technique in a query based on the spatial abstraction method described above, the ink analysis tool 1901 can efficiently obtain the required data, even for large documents. It becomes possible to analyze. However, it should be noted that the application 1701 can simply control the content of the document independent analysis context object 1903 by not exposing the content to an analysis tool, if desired.

Also, while the ink analysis tool 1901 creates the document independent analysis context object 1903, the software application 1701 should not change the analysis context object 1709 (and other threads in the application). And / or other programs are not allowed to change the analysis context object 1709 during this time period. That is, the software application 1701 should not enter new data into the document 1703 during this period, which is not allowed to other threads and / or applications. However, the process of creating the document independent analysis context object 1903 is relatively fast, which will typically not have a significant impact on the operation of the software application 1701. However, as needed, the results of entering data into document 1703 may be cached and entered into the system after document independent analysis context object 1903 is generated.

Once the document independent analysis context object 1903 is created, all subsequent analysis operations on the unanalyzed ink and / or other data may be performed on the document independent analysis context object 1903 rather than on the analysis context object 1709. Thus, the software application 1701 may continue to perform normal operations without being delayed or interrupted by the analysis of unanalyzed ink and / or other data. The software application 1701 can input new electronic ink data 1705 into the document 1703 without interfering with the analysis of the unanalyzed ink in the document independent analysis context object 1903.

Thus, after the document independent analysis context object 1903 is created, in step 1605 the ink analysis tool 1901 creates a separate analysis thread to analyze the ink in the document independent analysis context object 1903. Thus, at step 1607, the software application 1701 can resume control of the main processing thread and continue to perform normal operations. Then, in step 1609, the ink analysis tool 1901 provides the document independent analysis context object 1903 to the first analysis process for analysis. For example, as shown in FIG. 20, the ink analysis tool 1901 may pass the document independent analysis context object 1903 to a parsing process 2001 for parsing using an analysis thread. The parsing process includes one or more classification processes (e.g., ink and / or other data, such as ink text, ink pictures, tables, charts, graphics, images, music, mathematics, etc.) without departing from the invention. Processing to classify into a variety of different types, such as pictures with special symbols (such as electrical schematics with resistors, capacitors, etc.). In some examples of the invention, it should be noted that the ink analysis tool 1901 may make a copy of the document independent analysis context object 1903. As described in more detail below, a copy of the original form of the document independent analysis context object 1903 can be used to adjust the analysis results of the analysis process using the current state of the document 1703. .

After the first analysis process analyzes the ink and / or other data in the document independent analysis context object 1903, the first analysis process returns the analysis results to the ink analysis tool 1901, which in turn returns the analysis results to step 1611. Notifies the software application 1701. For example, as shown in FIG. 21, when the first analysis process is a parsing process 2001, the parsing process returns the parsing results 2101 to the ink analysis tool 1901, and then (for example, By firing an event, a reference to the analysis results 2101 can be passed to the software application 1701.

As will be described in more detail below, in some examples of the invention, the analysis results 2101 may be a document independent analysis context object 1903 that was originally provided for the analysis process, but by the first analysis process. It may be changed to include the generated new information (eg parsing results in this example). However, in still other examples of the invention, analysis results 2101 may be a modified copy of document independent analysis context object 1903. By including the new information generated by the first analysis process in a copy of the document independent analysis context object 1903, the original document independent analysis context object 1903 can be used, for example, by a reconciler, and And / or reserved for use in other analytical processing described above.

22 shows an example of how the parsing process (eg, parsing process 2001) may transform the data tree 1801 to show layout changes generated by the parsing operation. As shown in this figure, the parsing process 2001 has determined that the unanalyzed ink stroke indicated by the stroke data 1823 is part of the word indicated by the word node 1813. Thus, the parsing process disassociates the root node 1803 from its stroke data 1823 and associates it with the word node 1813 instead.

The parsing process also determined that the ink strokes represented by the unclassified stroke data 1825 and 1827 are part of a new word in the line represented by the line node 1809 not identified above. Thus, parsing process 2001 generates a new word node 2201 and associates the line node 1809 with the word node 2201. Parsing process 2001 then associates the new word node 2201 with the stroke data 1825, 1827. Thus, parsing results 2101 describe the relationships between unanalyzed ink strokes that were not previously analyzed and other ink strokes (or other document elements) identified by parsing process 2001. Further, in addition to showing the relationship changes determined by parsing process 2001, parsing results 2101 may also include classification information determined by parsing processes 2101. For example, each instance of the ink stroke data 1823-1827 may be modified to classify the corresponding ink stroke as text ink stroke rather than picture ink stroke. In at least some examples of the invention, the parsing results 2101 may be displayed in the document independent analysis context object 1903, if necessary.

As described above, while the first analysis process analyzes the document independent analysis context object 1903, the software application 1701 may include new ink data 1705 and new non-ink data 1707. Data can be entered into the document 1703. Thus, the results of the first analysis process may no longer be applicable to the current state of document 1703. For example, the parsing process may determine that the ink stroke is associated with the word, but the user may delete all of the ink stroke (eg, with all or more associated words) from the document 1703. Thus, in step 1613 the analysis results 2101 should be adjusted to the current state of the document 1703. An exemplary adjustment process is shown schematically in FIG. 23, which will be described in more detail below.

As some examples of the invention, the software application 1701 may manually adjust the analysis results 2101 to the current state of the document 1703. That is, the software application 1701 can look closely at the analysis results 2101 to determine whether the results are related to document elements in the current document 1703. However, in other examples of the invention, the ink analysis tool 1901 may analyze the analysis results 2101 in the current analysis context object 1709 (ie, the analysis context object 1709 reflecting the current state of the document 1703). It can provide the adjustment function to adjust with. As described in more detail below, the confrontation between the analysis results 2101 from the first analysis process and the analysis context object 1709 for the current state of the document 1703 by the adjustment function of various embodiments of the present invention. (conflict) or "collision" will be identified. The ink analysis tool 1901 then updates the analysis context object 1709 based on the analysis results in step 1615 and, from step 1617, the new document independent analysis context object (from the adjusted analysis results of the analysis context object 1709). 1903).

While the ink analysis tool 1901 adjusts the analysis results 2101 to the current analysis context object 1709, the software application 1701 is not allowed to change the analysis context object 1709 (and the analysis context). It should be noted that the object 1709 should not be allowed to be changed by the application and / or other threads in any other application. That is, software application 1701 should not enter new data into document 1703 until adjustments are complete, and / or allow other threads and / or applications to enter new data into document 1703. You should not. As with the process of first generating the document independent analysis context object 1903, the coordination process is relatively fast and will typically not have a significant impact on the operation of the software application 1701. However, if desired, the attempted user input may be cached and entered after the adjustment process is completed.

After the ink analysis tool 1901 adjusts the analysis results 2101 to an analysis context object 1709 for the current state of the document 1703, the ink analysis tool 19901 selectively converts the adjusted analysis results to another one. Can be provided to a second analysis process for analysis. For example, as shown in FIG. 24, the ink analysis tool 1901 can handwriting adjusted parsing results 2401 (eg, text, music, mathematical information, or other specific data types) for recognition. To the ink recognition process 2003 for recognition. In particular, in step 1619 the ink analysis tool 1901 will re-create a separate analysis thread for performing the second analysis process. Since the ink analysis tool 1901 creates such a separate thread for executing the second analysis process, the software application 1701 can resume normal operation in step 1621 and even include new electronic ink data 1705. It can also accept new input data.

Next, at step 1623, the ink analysis tool 1901 provides the results 2401 adjusted from the first analysis process to the second analysis process. Once the second analysis process has begun, the application 1701 can begin another execution of the first analysis process for the new unanalyzed ink. Thus, multiple analysis processes can be executed simultaneously, so that while the second analysis process is performed on the results of the first analysis process, the application 1701 performs the first analysis process to prepare for the next set of analysis results. You can restart it.

Advantageously, executing the analysis processes in parallel can improve the operation of the software application 1701. For example, the parsing analysis process is typically faster than the recognition analysis process, and the parsing analysis results may be used to implement an accurate selection operation and a space-insertion operation without using the recognition analysis results. Thus, because other analysis processes can be executed in parallel, the new ink added to the document 1703 needs to wait for the recognition analysis process to be completed for the old ink data before it is selected or the space is correctly inserted therein. There is no. In addition, various embodiments of the present invention provide asynchronous ink to free parser and recognizer algorithm developers and application developers from multithreading problems, while improving maintainability and simplifying the development process for both groups. It realizes the analysis and allows for changes in the interactions between the analysis processes without changing the application.

As described above, the ink analysis tool 1901 is adjusted by generating a new document independent analysis context object 1903 from the analysis document object 1703 after it has been updated to include relevant portions of the analysis results 2101. Can provide results. However, in other examples of the invention, the ink analysis tool 1901 can simply update the original document independent analysis context object 1903 with analysis results 2101 to reflect the current state of the document 1703. . In addition, the ink analysis tool 1901 may simply further provide related ink data to be analyzed to the second analysis process.

Therefore, various embodiments of the present invention adjust the results of the preceding analysis process to the current state of the document 1703 before executing the subsequent analysis process. However, it should be understood that this intermediate adjustment step may be omitted in some examples of the invention. In particular, in some examples of the present invention, the results of the foregoing analytical processing may be provided directly to subsequent analytical processing, without adjusting the results to the current state of document 1703. For example, if the ink analyzer process is provided separately from the ink layout process, ink classified as handwritten text may be provided to the ink layout analyzer process without an intermediate adjustment step.

In any arrangement, after the second analysis process 2003 has completed the analysis of the adjusted results 2401 from the first analysis process, the second analysis process 2003 may display the results as shown in FIG. 25. 2501 is returned to the ink analysis tool 1901. The ink analysis tool 1901 then informs the software application 1701 of the results 2501 of the second analysis in step 1625.

As described above, while the second analysis process analyzes the adjusted results 2401 of the first analysis process, the software application 1701 may write new data including the new electronic ink data 1705 in the document 1703. You can type in Therefore, the second analysis processing results 2501 can no longer be applied to the current state of the document 1703. Thus, the results 2501 of the second analysis process 2003 should also be adjusted to the current state of the document 1703 at step 1627. An exemplary adjustment process is shown schematically in FIG. 26, which will be described in more detail below.

Again, as some examples of the invention, the software application 1701 may manually adjust the second analysis results 2501 to the current state of the document 1703. Alternatively, as other examples of the present invention, the ink analysis tool 1901 may provide an adjustment function to adjust the analysis results 2101 to the current analysis context object 1709. This adjustment function will identify an conflict or “collision” between the analysis results 2501 from the second analysis process and the analysis context object 1709 for the current state of the document 1703, followed by the ink analysis tool 1901. Will update the analysis context object 1709 based on the analysis results in step 1629. This process can then be repeated as necessary to analyze the new ink input or ink while the analysis processes are being executed.

From the foregoing, it is possible for the electronic ink to be analyzed asynchronously from the software application 1701 by ink analysis techniques in accordance with various embodiments of the present invention, wherein the software application 1701 is a new electronic ink during the analysis process. It is allowed to continue to accept input data that includes. Although only the two analytical processes are described in the above description, it should be understood that any number of analytical processes can be executed simultaneously in parallel. For example, various implementations of the invention may utilize first parsing processing, second parsing processing, and recognition processing. In this way, once each analysis process is completed, the results can be transferred to the next analysis process, and the first analysis process can again be executed simultaneously and in parallel with the newly input ink.

In addition, in various embodiments of the present invention, one or more analytical processing may be performed sequentially. For example, with some software applications, a user can specify a particular language (such as English or Japanese) for ink strokes or combinations of strokes. This language specification may then be included in the analysis context object 1709 and document independent analysis context object 1903. Based on the language designation, the ink analysis tool 1901 routes the electronic ink designated in the first language to a recognition process tailored to the first language, and subsequently passes the electronic ink designated in the second language to the second language. You can route to another recognition process that fits. In this same manner, input data classified and / or designated in other specific types may be sent to other specific identifiers such as musical identifiers, electrical symbols, mathematical symbols, flowchart shapes, graphical shapes, and the like. .

In addition, the analysis processes need not be part of the software application 1701, so any desired analysis process can be used to analyze the electronic ink in the software application 1701. For example, a software application developer may develop and use a parsing process or recognition process that is particularly suitable for analyzing the electronic ink input expected for the software application 1701.

Still other variations of these techniques can be used by various implementations of the invention. For example, in some situations, it may be undesirable for the software application 1701 to maintain an internal document tree that reflects relationships between various document elements in the document 1703. Instead, the software application 1701 may be solely for using simple state information regarding the electronic ink input. In such software applications 1701, the software application 1701 may omit generating the analysis context object 1709 that reflects the current state of the entire document 1703. Rather, software application 1701 can generate a particular-purpose analysis context object 1709 that only includes information about the particular ink input that the software application 1701 desired to analyze. By using the specific-purpose analysis context object 1709, the software application 1701 can use the desired analysis processes to analyze the ink in the document, while the internal data structure corresponds to the state of the document 1703. The complexity of maintaining is avoided.

In addition, although the above description has described asynchronous analysis of electronic ink, it should be understood that various embodiments of the present invention also allow the ink analysis tool 1901 to analyze the ink synchronously. For example, some implementations of the invention allow the software application 1701 to instruct the ink analysis tool 1901 to perform an immediate and synchronous analysis of the electronic ink. In addition, the synchronous analysis may be for the ink in the entire document 1703 or only a specific area of the document 1703.

Ink Analysis Tool

As discussed above, the ink analysis tool 1901 performs various functions to facilitate processing of the electronic ink. For example, in accordance with various implementations of the invention, the ink analysis tool 1901 receives information from a software application 1701 that designates one or more regions within a document 1703 having ink data to be analyzed, and that The software application 1701 is queried to obtain information regarding the areas, and then constructs a document independent analysis context object 1903 based on the information. The ink analysis tool 1901 also provides a document independent analysis context object 1903 to one or more analysis processes for analysis, and then adjusts the results of the analysis process to the current state of the document 1703. Various processes may be performed using separate threads (as described above), through a thread pool, through separate machines in a cluster, or as separate processes, without departing from the invention. Or in any other suitable manner or desired manner.

According to other examples of the invention, various software objects and tools may be used to implement the ink analysis tool 1901 and perform these functions. However, in some examples of the invention, the ink analysis tool 1901 may be implemented as an application programming interface (API). In particular, in accordance with some examples of the present invention, the ink analysis tool 1901 may include software object routines and associated information that may be called by the software application 1701 as needed to analyze the ink in the document 1703. It can be implemented as a group.

As will be appreciated by one of ordinary skill in the art, an application programming interface may be organized into "classes" or categories of software objects. An "object" is a collection of storeable state values and executable actions. In particular, an object may maintain different types of state values, referred to as "fields" and "properties," that may be associated with or retrieved by the object. In an external software application or another software object, these attributes are read only (identified by the use of the term "{get;}"), write only (term "{set;} "Identified by the user of", or both readable and writable (identified as "{get; set}"). An object can often execute tasks or "methods" in a request or "call" of a software application or another object. Therefore, the application programming interface may include various objects that another software application can instruct to perform specific tasks or provide specific information on request.

It should be appreciated that various embodiments of the present invention may be implemented that require the software application 1701 to provide only the minimum amount of information needed to complete the desired analysis process. For this reason, some of the attributes described in detail below may be described as only read-only (i.e., "{get;}") or write-only (i.e., "{set;}") attributes, but various alternatives of the invention Implementations may be both readable and writable. For example, an attribute that defines "hints" to be used by the analysis process to analyze the electronic ink may be identified as a read-only attribute in situations where the software application 1701 should not be intended to use the hints. Can be. This allows objects created by the ink analyzer to read at least these attributes, and the values of these attributes are default values that are null (ie, the software application 1701 does not want any "hints"). Is determined. However, if the software application 1701 wants to use the hints for ink analysis, this attribute is read to allow the software application 1701 to change the value of this attribute from the null value to specify the desired hints. It will be both possible and recordable.

According to various embodiments of the present invention, an API that embodies the implementation of ink analysis tool 1901 (hereinafter referred to as ink analysis API) may include two core classes. The first class will be referred to as the "Analysis Context" class, and the second class is the "Ink Analyzer" class. Components of the "Analysis Context" class are used to create the analysis context object 1709. Thus, in various implementations of the invention in which the software application 1701 creates and maintains its analysis context object 1701, one or more components of this class may be omitted from the ink analysis tool. Instead, one or more of these components may be implemented by the software application 1701 itself.

The components of the class "Ink Analyzer" can then be used to create and use an object that provides the document independent analysis context object 1903 to the analysis process, determining when the analysis results were generated and documenting the analysis results 1703. Can be used to adjust the current state of Hereinafter, these and other classes that can be included in the ink analysis API according to various embodiments of the present invention are described in detail.

Returning to the analysis context class first, this class is implemented by the host application 1701 to create the analysis context object 1703, which acts as a proxy view in the internal document tree of the software application 1701. do. Alternatively, as described above, when an application implements its own analysis context object, one or more components in this class may be omitted from the ink analysis tool 1901. As discussed above, analysis context object 1703 includes all unanalyzed ink data, and analysis context object 1703 is used to identify which unanalyzed ink data should be analyzed. The analysis context object 1703 also contains information about the ink that has already been analyzed. This already analyzed ink can now be used to determine how the unanalyzed ink should be analyzed and can be changed while analyzing the unanalyzed ink on its own. The analysis content object 1703 also includes information about non-ink content of the document 1703, which is used to properly classify the ink as annotations for non-ink content.

According to various embodiments of the invention, the analysis context class includes a constructor that, when called by the software application 1701, creates an analysis context object 1709. This class may also include various attributes for the analysis context object 1709 that include an attribute named "Dirty Region {get;}". The dirty area attribute defines the portion of the document (and portion of the analysis context object 1709) that contains the unresolved ink data. In various embodiments of the present invention, the dirty area may define a disjoint area. The dirty region is specified as an AnalysisRegion, which will be described in more detail below, but may simply be a collection of rectangular regions within a document. If the dirty area is empty, no analysis will occur. This class further includes an attribute for the analysis context object named "Margins {get;}", which declares areas of the page in document 1703, which are considered margins. For example, an analytical process that analyzes the layout and determines the classification of the electronic ink can use these attributes to help determine the classification of electronic ink (eg, ink on a margin) that is non-ink content.

In addition, the analysis context class may include an attribute named "Rootnode {get;}", which identifies the best context node or root context node in the analysis context object 1709. As mentioned above, this root node includes all other context nodes for a given analysis context object 1709 as child context nodes. It should be noted that as various embodiments of the present invention, the root context node should be of context node type "Root". In the examples of the invention in which the application 1701 implements its own analysis context object 1709, the analysis context object 1709 may have other context nodes, such as siblings of the root context node, but with ink It should also be noted that the components of the resolver class may be limited to considering the context nodes included by the root context node.

The analysis context class may further include the attribute "Analysis Hints {get;}", which returns an array of analysis hint objects set by the software application 1701. As discussed in more detail below, analysis hint objects may include any type of information that may aid in the analysis process. Such information may include, for example, guides or word lists as factoids. It may also contain information that sets the language to be used for analysis, or provide any kind of guidance for parsing processes such as identifying ink as a list, table, shape, flowchart, connector, container, or the like. It may include information specifying the unanalyzed ink only as handwritten text or only as a picture.

In addition to these attributes, the analysis context class may also include several methods that can be called by the software application 1701, for example, for the analysis context object 1709 to perform a task. For example, the analysis context class may include a method named "FindNode (Guid id)". Each node of the analytics context object 1709 has a globally unique identifier (ie, a GUID), and this method will be located at the node specified by the call somewhere within the analytics context object 1709. Since the method can be called from many time critical operations, this lookup method should be implemented as efficiently as possible.

Like the analysis context class, the ink analyzer class also defines a public constructor according to various properties that allows the software application 1701 to create an instance of the class (ie, ink analyzer object). For example, it may include an attribute named "User Interface Context {get; set;}" which defines a processing thread in which the results of the analysis process are returned. This property allows the results to be synchronized to another object. For example, if this is set to its main form, parser results will be fired to the application main thread. It may also include an attribute "AnalysisOptions AnalysisOptions {get; set '}" specifying various criteria that can be used in the analysis process. These criteria may, for example, enable text recognition, enable the use of tables, enable the use of lists, enable the use of annotations, enable the use of connectors and containers. It may include doing.

The ink analyzer class will contain several methods. For example, such a class might contain a method named "AnalysisRegion Analyze ()". These methods start the synchronous analysis process. The document element data is passed to this method, which describes the current state of the document 1703 and represents which ink in the document 1703 needs to be analyzed. In some implementations of the invention, document element data can be provided as an analysis context object 1709 (ie, AnalysisRegion Analyze (Analysis Context)), as described above. Alternatively, individual ink strokes may use references to strokes (ie, AnalysisRegion Analyze (Strokes)) or ink analyzer objects (eg, InkAnalyzer.Strokes {get; set}) without attributes passed to the analysis method. It can be referred to as an attribute of and passed to the analysis process.

Once the analysis process is complete, this method will return a reference to the document independent analysis context object that has changed to include the results of the analysis process. The method also returns an AnalysisRegion value (described in detail below) that describes the area within the document from which the results were calculated.

The ink analyzer class may also include a method labeled "AnalysisRegion Analyze (AnalsisContext, waitRegion)". This method is identical to the synchronous Analysis Region Analyze () method described above, but only has the ink analyzed when results are needed in a particular waitRegion region. In particular, a call to this method will identify the analysis context object 1709 for the document 1703 and the area of the analysis context object 1709 (referred to as "waitRegion") that the analysis process should analyze synchronously. will be. In various embodiments of the present invention, all other areas of the analysis context object 1709 will be ignored if the analysis process does not need to analyze the content in those areas to complete the analysis of waitRegion. As mentioned above, the analysis context object 1709 passed to this method includes an attribute called "DirtyRegion" that describes the areas of the document 1703 that require analysis. By specifying a particular waitRegion, the software application 1701 can obtain analysis results for one particular area of interest more quickly than having all ink data in the document 1703 analyzed.

When any of these analysis methods are called, all available analysis processing will be executed. Also, since these analysis methods are synchronous invocations, the reconciliation process does not need to be executed at its end and will not execute the event fire upon completion.

The ink analyzer class also includes a method labeled "BackgroundAnalyze (AnalysisContext)". This method initiates the specified analysis operation, but also proceeds to a separate background analysis thread. Therefore, this method will return control to the main processing thread almost immediately after the actual analysis operation is completed in the background. In particular, this method will return a value of "true" if the analysis process has been initiated successfully. Again, the AnalysisContext value passed to the method identifies the analysis context object 1709 for the document 1703 and indicates which ink in the document 1703 needs to be analyzed. Once the analysis operation on the background thread is completed, a Results event is raised to allow the software application 1701 to access the results. The event includes results and an adjustment method, which is used to integrate the results back into the analysis context object 1709 for the current state of the document 1703 when the results are returned.

It should be noted that each of these three analysis methods in turn calls the method "Clone" in the "Analysis Region" class, as discussed in detail below. Using the "Clone" method, these analysis methods create an independent document analysis context object that is subsequently changed by the analysis process to show the analysis results.

The ink analyzer class is also marked with "Reconcile (AnalysisContext current, AnalysisResultsEventArgs resultArgs)" which is called by the software application 1701 after receiving the result event generated by calling the BackgroundAnalyze (AnalysisContext) method. It can contain a method. The adjustment method compares the analysis results included in the document independent analysis context object with the current version of the analysis context object 1709 maintained by the software application 1701. This method identifies nodes that need to be removed or added to the current version of analysis context object 1709, and recognizes results, location, ink strokes associated with the node or the results of the analysis operation, which are attributes of the existing node. Identify any of any other data associated with the. This method also writes these identified changes to the current version of the analysis context object 1709. This method is sensitive to context node order, such as the order of word context nodes on a line context node.

The analysis results (ie, the value of the attribute AnalysisResultsEventArgs) are passed back to this method because they include the public results structure and the private results structure. The public structure is returned so that the software application 1701 can preview the changes that occur in the adjustment phase. The private structure is included to prevent the software application 1701 from changing any of the analysis results before the adjustment process.

The ink analyzer class may also include methods labeled "Recognizers RecognizersPriority ()" and "SetHighestPriorityRecognizer (recognizer)". If the ink needs to be recognized, an appropriate recognizer will be used based on the languages and capabilities. Thus, the Recognizers RecognizerPriority () method returns recognition processes in the order in which they will be evaluated by the ink analyzer object. The order is determined per system depending on whether recognition processes are available, but can be overridden for software application 1701 by calling SetHighestPriorityRecognizer (recognizer) on the ink analyzer object. The ink analyzer will list this ordering list until a suitable recognizer is found. The SetHighestPriorityRecognizer (recognizer) method raises the priority of the recognition process. By raising the priority of a particular recognition process, the recognition process will be used if it meets the required language and capabilities of the current recognition operation. In essence, the SetHighestPriorityRecognizer (recognizer) pushes the specified recognition process to the top of the list returned by the RecognizersPriority method.

The ink analyzer class also includes a method represented by "AnalysisRegion Abort ()", which can take an analysis context object as a parameter. This method allows the foreground or background analysis to terminate early. These methods return an analysis region that describes the region that was analyzed before the break. Therefore, if the software application 1701 intends to continue the analysis operation later, this area may be merged into the dirty region DirtyRegion of the analysis context object 1709 for the current state of the document 1703. In addition, the ink analyzer class may include an event represented by "AnalysisResultsEventHandler" which actually fires the ink analyzer object frequently. In particular, such an event may fire between analysis processes, at least once every 5 seconds. This event can be used to provide an application 1701 with an update regarding the status of an ongoing asynchronous analysis process (or processes).

The ink analysis API may also include classes in addition to the analysis context class and the ink analyzer class. For example, the ink analysis API according to various embodiments of the present invention may include a context node class. This class may include various components related to the document independent analysis context object and the context nodes that make up the analysis context node 1709, such as the attribute denoted as "ContextNodeType Type {get;}". As will be understood by one of ordinary skill in the art, each context node has a type, and there is a specific set of rules that each type must adhere to. This includes, for example, rules such as what types of child context nodes are allowed and whether the strokes can be directly associated with the context node or only through the child context nodes.

Possible types of context nodes may be defined with a ContextNodeTypes enumeration, for example the following types: ink annotation node representing ink data annotating non-text data; An ink picture node representing ink data forming a picture; An ink word node representing ink data forming a word, a line node comprising one or more ink word nodes, and / or text word nodes for words forming a line of text; A list item node, which may include nodes, such as expected paragraphs, images, in the list; And a list node comprising one or more list item nodes each listing an entry in the list. Node types may also include non-ink picture nodes representing non-ink picture images; An object node representing data not covered by other values of the context node type enumeration; A shorting node comprising one or more line nodes corresponding to the lines forming the shorting; A picture or image node representing a picture image; A root node that serves as the best node in the analysis context object; A table node including nodes representing items constituting the table; A text box node representing a text box; Text word node; And an Unclassified Ink node that still corresponds to unclassified ink data. The node types may also be group nodes for groups of other nodes, ink bullet nodes for bullet items, row nodes for document elements presented in rows of the table, and documents presented in cells of the table. It may include cell nodes for elements.

The Context Node class may also include an attribute named "GUID Id {get;}", which is the global single identifier for the current context node. To allow access to any desired context node, each context node within a single analysis context object must always have a single identifier. This class may also contain an attribute named "AnalysisRegion Location {get;}" that identifies the location in the document space where the relevant context node is actually located. As mentioned above, the AnalysisRegion is a two-dimensional structure that groups together one or more possible disassembly rectangle structures. This class may also include an attribute named "StrokeCollection Strokes {get;}", which identifies ink strokes associated with the associated context node. In various embodiments of the present invention, only leaf context nodes (such as words, pictures, and bullet nodes) are allowed by the ink analysis API to have strokes. Software application 1701 may use these attributes to refer to the strokes at the leaf node level by all ancestor context nodes (e.g., the root node may refer to a stroke reference for all strokes in the associated analysis context). Will be included in the object).

In addition, this class may include an attribute named "ContextNode ParentNode {get;}" that identifies the parent context node that contains the associated context node. In various embodiments of the present invention, context nodes are always created for the root context node to rely from the parent context node, which is a static number of analysis context objects. This class may also contain an attribute "ContextNode [] SubNodes {get;}" that identifies all context nodes that are direct children of the associated context node. That is, this attribute will only identify child context nodes one level down in the analysis context object. For example, the value of the attribute for such a paragraph context node will only identify the line context nodes included by the paragraph node and not the word context nodes that are children of the line context node.

This class is also represented as "RecognitionResult RecognitionResult {get;}", which provides recognition results calculated by the relevant recognition analysis process or processes, since the recognition result can represent one or more lines of text in one or more languages. It can contain attributes. The recognition result is set by the recognition analysis process and although the recognition result attribute (described in more detail below) used to generate the recognition result object may be set to only one level of the context node tree to avoid duplication of data. In turn, all context nodes within the document independent analysis context object are available. If a node does not have recognition data associated with it, it will either merge the recognition results of all its subnodes or extract the recognition results from its parent. This class may also contain an attribute denoted as "Stream RecognitionData {get; set;}", which is a persistent form of recognition result value. Again, the recognition analysis process generates a stream recognition data value set to the relevant context node. The recognition result object is then constructed based on these values.

The context node class may further include an attribute denoted as "ContextLink [] Links {get;}", which provides an array of context link objects. The context link object describes the replaceable relationship between two context nodes. Context nodes typically have a sub-child relationship with other context nodes, while context links allow for an alternative relationship between context nodes. For example, a context link may be a connection between two context nodes, anchoring one context node to another context node, suppressing one context node by another context node, or to a software application 1701. It can allow the desired link type defined by Contextual links can be added to this array by calling the add link method, which is discussed in more detail below. Similarly, contextual links can be removed from this array by invoking the link delete method discussed in more detail below.

In addition, this class may include attributes "IsContainer {get;}" and "IsInkLeaf {get;}". The property IsContainer {get;} has a value of "true" if the related context node is not a leaf context node (ie, if the related context node contains child context nodes and thus is considered a container context node), otherwise it is "false". (false) "value. The IsInkLeaf {get;} attribute has a value of "true" if the current context node is not a container context node, otherwise it has a value of "false". In other words, if the current context node does not contain any child context nodes, it is considered a leaf context node. In various embodiments of the present invention, an ink leaf context node is expected to contain references to stroke data, but container context nodes do not have this limitation. Container context nodes may or may not refer to the stroke data as specified by software application 1701.

The context node class may also include the attribute "Rect RotatedBoundingBox {get; set;}". The value of this attribute is calculated by the layout and classification analysis process. If ink data associated with an associated context node is written at an angle, the bounds for that context node will still be horizontally aligned. However, the value of the RotatedBoundingBox attribute will be aligned at the angle at which the ink data associated with the associated context node is described. This class may also include an attribute "ReClassifiable {get;}" that informs the ink analyzer whether it is allowed to change the values of the associated context node.

In addition to these attributes, the context node class may also include several methods. For example, this class includes a method that is labeled "ContextNode CreateSubNode (ContextNodeType type)". These methods allow the creation of child context nodes of a particular type. In various embodiments of the present invention, this method may only allow valid child types of the associated context node to be created, thereby preventing malformed data structures from being created. For example, this method may only allow line context nodes to generate ink words and text word context nodes. This class may also include a method labeled "void DeleteSubNode (ContextNode node)", which deletes the referenced child context node from the associated analysis context object. However, it should be noted that in various embodiments of the present invention, if the referenced context node includes strokes or child context nodes, this method should fail. Also, if the reference context node is not a direct child of the associated context node, this method fails. When the software application 1701 implements its own analysis context object 1709 and in turn uses this method, a non-empty context to prevent malformed data structures in the analysis context object 1709. Note that context nodes that are not direct children of nodes or related context nodes should not be deleted.

This class may also include the method "ContextNode [] HitTestSubNodes (AnalysisRegion region)", which returns an array of context nodes located within that particular region. However, it should be noted that only the immediate child nodes of this element are returned, not all descendants. The area is defined by the analysis area object, which may be a collection of one or more rectangles, as mentioned above. According to various embodiments of the present invention, if any portion of the location of the context node intersects a particular area, the context node will be returned to the array. This method is used, for example, to create a document independent analysis context object and adjust its analysis results to an analysis context object corresponding to the current state of document 1703. Therefore, this method is frequently called and should be optimized for fast, repeated access by the ink analyzer object.

The context node class also includes a method labeled "MoveStroke (Stroke stroke, ContextNode destination)". This method moves the association of a stroke from one leaf context node to another. In various embodiments of the present invention, this method is used only between leaf context nodes. It may also include a method labeled "MoveSubNodeToPosition (int OldIndex, int NewIndex)", which reorders the relevant context nodes for sibling context nodes. For example, if document 1703 has three words on a line, for example word 1, word 2, word 3, their order is implied by the array of subnodes returned from the parent context node. imply). This method allows their order to be changed so that word 1 becomes the last word on the line by moving the context node for word 1 from position 1 to position 3 with respect to the associated context node.

In addition, this class may include a method labeled "AddLink (ContextLink link)", which adds a new context link object to the current context node. In various embodiments of the present invention, the context link object should include a reference to the relevant context node in the order of the context link successfully added to the array of context links associated with the relevant context node. It may also include a method labeled "DeleteLink (ContextLink link)". This method deletes or removes a specific context link object from the array of context links for the relevant context node. In various embodiments of the present invention, this method always successfully completes the call even when the context link is not present in the array of context links associated with the associated context node.

The ink analysis API also includes an analysis hint class. In many of the classes described above, the analysis hint class includes a constructor, denoted "AnalysisHint ()", which initializes the analysis hint object to an empty state. This class can also contain many properties, including the property labeled "AnalysisRegion Location {get;}". This attribute specifies a location within document 1703 (as an analysis area) to which an analysis hint is applicable. For example, if the document 1703 is a free form note with a title portion at the top of the page, the application 1701 analyzes the title area to specify that horizontal lines of ink are expected to be within that area. You can set a hint. Analysis hints will help to increase the accuracy of the analysis process.

This class will also contain an attribute named "string Factoid {get; set;}" that specifies a particular "factoid" to be used for the location in the document 1703 where analysis hints are applicable. Can be. As is known to those of ordinary skill in the art, factoroids are involved in the recognition process regarding the expected use of ink data (e.g., regular text, numbers, zip codes, filenames and web URLs). Provide hints. This class is called the "RecognizerGuide Guide" {get; set;} "and" OverrideLanguageId {get; It can contain attributes named set;}. Recognizer Guide Guide {get; set;} attribute specifies a recording guide to be applied at a location in the document 1703 where analysis hints are applicable. For example, recording the guide helps the user to improve the accuracy of the recognizer analysis process by specifying and notifying the user of a recognizer analysis process that records lines or characters. OverrideLanguageId {get; set;} attribute specifies a language hint for the document 1703 to which the analysis hint is applicable. Setting a language hint causes the ink analyzer object to use the specified language instead of the language specified on the context node.

This class may also include an attribute named "PrefixText {get; set;}", which specifies the text written or typed earlier in the line of ink to be recognized. This class also includes an attribute denoted as "RecognitionModes RecognitionFlags {get; set;}", which specifies the specific types of modes that the recognition process should relate to a location in the document 1703 where analysis hints are applicable. This class also specifies an attribute named "SuffixText {get; set;}" that specifies text written or typed before the line of the link to be recognized, and a specific set of words that should be used by the recognition analysis process. It may include an attribute named "WordList WordList {get; set;}. If the expected recognition results are known before the user has the input data actually recorded, for example, inside a medical form. A word list can be used, such as a list of medical terms expected to be recorded.

This class may also contain an attribute named "WordMode {get; set;}". If this value is "true", the analysis process will bias itself to return a single word for the entire analysis area. Also, if "true", it will contain an attribute named "Coerce {get; set;}", which will force the analysis process to limit the result to any factoroid or wordlist set in the relevant hint. Can be. This class may also contain an attribute named "AllowPartialDictionaryTerms {get; set;}". If this attribute value is "true", the recognition analysis process will be allowed to return partial words from the recognition dictionary.

According to various embodiments of the present invention, the ink Kungseok API may further comprise an analysis region class. Such a class may include, for example, a number of constructors for constructing an analysis domain object. For example, a first constructor for constructing an analysis region object having an area, a second constructor for constructing an analysis region object based on parameters for a two-dimensional rectangle, and constructing an analysis region object based on spatial coordinates It may include a third constructor. For example, the default constructor can create an empty area. This class can also contain many attributes. For example, this class may contain an attribute named "Rectangle Bounds {get;}" that receives a bounding rectangle for the analysis region, and an "IsEmpty {get; } ", And an attribute named" IsInfinite {get;} ", indicating whether or not the associated analysis region is set to infinite.

This class also contains a number of methods, such as the method named "AnalysisRegion Clone ()", which duplicates the associated analysis region object. This class may also include a method named "Equals (AnalysisRegion otherRegion)", which tests whether a specified analysis region object (referred to as other region) is equal to the related analysis region object. This method returns a value of "true" if the interior of the specified analysis area object is the same as the interior of the associated analysis area object; otherwise it returns a value of "false".

This class may further include a method "Intersect (AnalysisRegion regionToIntersect)" that crops the associated analysis region object for the specified analysis region. Therefore, the resulting analysis region object will contain only regions that overlapped or traversed the specified analysis region. This class may also include a method named "Intersect (Rectangle rectangle)" that crops the associated analysis region object for a specified rectangle. Again, the resulting analysis region object will only contain regions that overlapped or traversed the specified rectangle. It may also include a method named "MakeEmpty ()", which initializes the relevant analysis area object to an empty interior, and a method called "MakeInfinite ()", which sets an area that is infinitely occupied by the related analysis area. . This is called a method called "Union (AnalysisRegion regionToUnion)", which specifies to join or add an Analysis Area object to the associated Analysis Area object, and "Union (Rectangle rectangle)", which binds the specified rectangle to the associated Analysis Area object. The method may further include various methods for combining or separating differently defined areas, such as being a method. In this method, the rectangle can be specified by the coordinate space for the associated analysis region object. Of course, this class may include a number of different methods for combining regions or extracting one region from another based on any desired definition of the regions.

The ink analysis API may also have a recognition result class. In many of the classes described above, the recognition result class may include one or more constructors. For example, such a class may include a constructor named "RecognitionResult (Stream lattice)", which constructs a recognition result object from a given recognition lattice. In various embodiments of the present invention, the recognition grid is a serialized form of results from the recognition process. This method may, for example, specify a recognition grid as a byte array to be used in the construction of the associated recognition result object. It may also include a constructor named "RecognitionResult (ContextNode node)", which constructs a recognition result object from a given context node. It can also include a constructor named "RecognitionResult (string Text, int StrokeCount)", which constructs a recognition result object from the specified text value, which in turn is associated with a specified number of strokes, and the recognition process is the actual hand. It may be used for correction if no alternative recognition value corresponding to the written ink data is provided. This class can also contain a constructor named "RecognitionResult (RecognitionResult leftRecognitionResult, RecognitionResult rightRecognitionResult)", which constructs a recognition result object by merging two existing recognition result objects together.

RecognitionAlternate TopAlternate, an attribute named "StrokeCollection StrokeCollection {get;}", which provides an array of stroke indices representing a collection of strokes contained in a single ink object, and the best alternative to recognition results. It may contain one or more attributes, such as an attribute named {get;} ". This class also recognizes an attribute and recognition named "RecognitionConfidence RecognitionConfidence {get;}", which provides a level of confidence (eg, strong, medium, or poor) for the selection of the best alternative of the current result from the recognition analysis process. It may include an attribute named "string TopString {get;}", which returns the best result string of analysis results from the analysis process.

The recognition result class also includes many methods, such as a method named "public RecognitionAlternateCollection GetAlternateCollectionFromSelection (selectionStart, selectionLength, maximumAlternates)" that specifies a collection of alternatives from the selection in the best result string of analysis results from the recognition analysis process. Each alternative corresponds to only one ink segment. The input parameters for this method specify, for example, a value specifying the start of the text selection in which the collection of alternatives is returned, a value specifying the length of the text selection in which the collection of alternatives is returned, and a maximum number of alternatives to return. It may contain a value. This method may then return a RecognitionAlternateCollection collection of alternatives from the selection within the best result string of the recognition result.

The recognition result class may further include a method named "RecognitionResults Merge (RecognitionResults left, string separator, RecognitionResults right)". This method creates a new recognition result object from a single string that becomes a flat lattice, adds a single string to the beginning and end of an existing recognition result object, or between two existing recognition result objects. Can be used to concatenate single strings. This class may also include a method named "ModifyTopAlternate (RecognitionAlternate alternate)", which specifies the recognition result to be changed to a known alternative. In some embodiments of the present invention, by default, the highest result string of the results of the recognition analysis process corresponds to the top alternate. However, this method can be used to specify that alternatives other than the top most alternative are used in the results of the recognition analysis process. If the new best alternative has a different segment than before, the modify top alternative method will automatically update the context nodes to reflect the changes.

Note that this method calls the GetAlternatesFromSelection method, discussed in detail below, to retrieve alternatives that can be used to change the recognition result. This class can also have a method named "Stream Save ()", which persists the associated RecognitionResults object in the form of a recognition grid. The recognition grid is a seriesed format for use in representing the results from the recognition process.

The ink analysis API may also have analysis options of the type listed. This type may include one or more fields that specify how ink data will be analyzed by the analysis process, such as a field named "const AnalysisOptions Default", which enables all options available for the analysis process. For example, this field may enable text recognition, use of tables, use of lists, use of comments, use of connectors and containers, and use of intermediate results. This type also enables the field named "const AnalysisOptions EnableAnnotations", which can enable and disable the detection of annotations, the field named "const AnalysisOptions EnableConnectorsAndContainers", which enables and disables the detection of connectors and containers, And enabling and disabling the return of analysis results to the software application 1701 between the use of different sequential analysis processes (eg, parsing process and subsequent cognitive processing), named "const AnalysisOptions EnableIntermediateResults". May contain fields. This type may also have a field named "const AnalysisOptions EnableLists" that enables and disables detection of lists and a field named "const AnalysisOptions EnableTables" that enables and disables detection of tables. This enumerated type may further include a field named "const AnalysisOptions EnableTables", which enables and disables text recognition analysis processing. However, it should be appreciated that if additional analysis processes (or other versions of the same analysis process) are available, this type may thus include additional analysis options.

The ink analysis API may also include an AnalysisResultsEventArgs class. This class may have a constructor named "public AnalysisResultsEventArgs ()" that contains the results of the analysis and creates a data structure that is returned to the software application 1701 when an AnalysisResult event is raised. This class may include an attribute named "InkAnalyzer InkAnalyzer {get;}", which identifies an ink analyzer (InkAnalyze) that performs an analysis process.

The API may also have a Line class, which may be useful to several types of operating systems that recognize the use of "Line" objects that represent geometric lines. This class can contain a constructor that creates a Line object, such as a constructor named "public Line (Point beginPoint, Point endPoint)". This class also contains an attribute named "public Point BeginPoint {get; set;}", which indicates the starting point of the line object, and an attribute named "public Point EndPoint {get; set;}", which indicates the ending point of the line object. It can contain multiple attributes, such as

In addition to these classes, the ink analysis API may also include a Recognition Alternate class. This class may include elements that represent possible word matches for segments of the ink compared to the dictionary of the recognizer. For example, this class is named "Line Ascender {get;}", which provides an ascender line of RecognitionAlternate objects that exist on a single line (the line is represented by two points). Attribute named "public Line Baseline {get;}", which provides a baseline of a recognition alternative object present on a single line, a descender line of a recognition alternative object present on a single line. line), which may contain an attribute named "Line Descender {get;}". This class may also contain an attribute named "RecognitionResult Extract {get;}", which provides RecognitionResults for the current recognition alternative object. This attribute can be used, for example, to extract a recognition result object for a word from the recognition result object for the line containing the word.

It also provides a property named "Line Midline {get;}", which provides a midline for recognition alternative objects that exist on a single line, that is, a collection of strokes contained in the ink object (ie, it is recognized Provide a StrokeCollection representing the strokes associated with the result (RecognitionResult), a attribute named "StrokeCollection Strokes {get;}", and a stroke contained in one or more ink objects representing the strokes associated with the RecognitionResult. It may contain an attribute named "StrokeCollection [] StrokesArray {get;}", which provides a collection of fields. This class also contains an attribute named "RecognitionConfidence RecognitionConfidence {get;}", which provides a confidence level (e.g., strong, medium or poor) determined by the recognition analysis process upon gesture or recognition alternative object recognition. can do. For non-line nodes, the lowest recognition confidence of the children of the relevant context nodes will be returned. It may also include an attribute named "string RecognizedString {get;}" that specifies an alternative result string. Thus, for any context node above the word context node, the results strings are concatenated together by this method. For example, a line node would contain a result string that in turn contains the results of all child nodes or word nodes. The paragraph node will then contain a result string that contains the results of all its child nodes or line nodes.

The Recognition Alternate class also uses a method named "StrokeCollection [] GetStrokeArrayFromTextRange (int selectionstart, int selectionlength)" that specifies, for example, a StrokeCollection from each ink object corresponding to a known text range. You can include one or more methods to include. This class may also contain a method named "StrokeCollection [] GetStrokesFromStrokesArrayRanges (StrokeCollection [] strokesArray)" that contains a known input collection of strokes and specifies the smallest collection of strokes that the recognizer can provide alternatives to. have. In particular, the strokes are returned by an array of ink objects that each include an array of stroke indices for the collection. It should be noted that the collection of ink strokes returned by this method may be matched to the input collection or larger if the input collection matches only a portion of the minimal recognition result that includes all the input strokes. This class contains a known input collection of strokes and a stroke corresponding to a known text range and method named "StrokeCollection GetStrokesFromStrokesRanges (StrokeCollection strokes)", which specifies the minimum collection of strokes that the recognizer can provide alternatives to. The method may further include a method named "StrokeCollection GetStrokesFromTextRanges (int selectionstart, int selectionlength)" for specifying a collection (StrokeCollection).

This class contains a method named "void GetTextRangeFromStrokes (ref int selectionstart, ref int selectionend, StrokeCollection strokes)" that specifies the minimum range of recognized text from which the recognizer can return an alternative containing a set of known strokes. It further includes a method named "void GetTextRangeFromStrokesArray (ref int selectionstart, ref int selectionend, StrokeCollection [] strokesarray)", which specifies the minimum range of recognized text from which the recognizer can return an alternative containing a set of known strokes. can do. It may have a method named "RecognitionAlternateCollection SplitWithConstantPropertyValue (GUID propertyType)", which returns a collection of alternatives that is the division of the alternative in which this method is called. Each alternative in the collection includes adjacent recognition segments that have the same attribute value for the attribute passed to the method. For example, this method can be used to obtain alternatives for dividing the original alternative into confidence level boundaries (strong, medium or poor), line boundaries or segment boundaries in the recognition result. It further includes a method named “byte [] GetPropertyValve (GUID propertyType)” that specifies the value of that alternative known property, such as the alternative recognizer reliability. However, not all perceptual analysis processes provide values for all attribute types. Therefore, this method provides data for the types supported by the relevant recognition analysis process.

The ink analysis API may also include a Recognition Alternate Collection class. Like many of the classes described above, this class may include a constructor named "RecognitionAlternateCollection ()" for creating a RecognitionAlternateCollection object. This class also provides access to a collection of alternative recognition values, an attribute named "Count {get;}", which provides collections included in the collection of many objects or alternative recognition values. Namely, an attribute named "IsSynchronized {get;}", which provides a value indicating whether it is synchronized with "thread safe", and can be used to synchronize access to a collection of alternative recognition values. It can contain many attributes, such as an attribute named "SyncRoot {get;}", which provides an object.

This class also creates a method named "virtual void CopyTo (Array array, int index)", which copies all the elements of the current collection of alternative recognition values into a particular one-dimensional array starting at the specified destination array index, and each generation One or more methods, such as a method named "IEnumerable IEnumerableGetEnumerator ()", a standard implementation of IEnumerable that allows callers to enumerate through each RecognitionAlternate in the collection of recognition values of this alternative. It may include. This class may also include a method named "RecognitionAlternateCollectionEnumerator GetEnumerator ()" that returns a RecognitionAlternateCollectionEnumerator that contains all the objects in the collection of recognition alternative values. This method can be used to retrieve each object, for example, from a collection of recognition alternative values.

The ink analysis API may additionally include Recognition Confidence enumeration and Recognition Mode enumeration, each of which may include one or more fields related to recognition analysis processing. For example, the Recognition Confidence class may include a field named "Intermediate" that indicates that the recognition analysis process is confident that the list provided with alternative recognition values is the correct result, and that the recognition analysis is present in a list provided with alternative recognition values. Includes a number of fields, such as a field named "Poor" indicating that there is no assurance that there is a result, and a field named "Strong" indicating that the recognition analysis process is convinced that the best alternative of alternative recognition values is correct. can do.

Similarly, the Recognition Mode class may include fields that specify how the recognition analysis process interprets the electronic ink data and thus how to determine the recognition result string. For example, this class is a field named "Coerce" that specifies that the recognition analysis process governs the recognition result based on the factoroid specified for the context, and that the recognition analysis process is a single line. It may include a field named "Line" that specifies handling of electronic ink data. This class also specifies that the field named "None" specifies that the recognition analysis process does not apply any recognition mode, and that the recognition analysis process treats the electronic ink data as forming a single word or character. , May contain a field named "Segment". This class may also contain a field named "TopInkBreaksOnly", which disables multiple segments.

The ink analysis API may also include a context link class, which defines how two context nodes can be linked together. The contextlink object itself represents two context nodes, the direction of the link and the type of the link. This class specifies an attribute named "ContextNode SourceNode {get;}" that specifies a source context node that is linked from another context node, and specifies the type of link relationship that exists between the source and destination context nodes. An attribute named "ContextLinkType LinkType {get;}", and an attribute named "CustomLinkType {get;}", which specifies that a custom link is being used. This situation will occur if the application decides to use the ink analyzer API's linking system to represent application specific links beyond what the API can recognize.

This class also contains an attribute named "ContextNode DestinationNode {get '}", which specifies a destination context node that is linked from another context node. There can be two constructors available to this class, which create a relationship between the existing source and destination context nodes.

This class may also contain an enumeration named "ContextLinkType enum", which defines the type of relationship shared by the two context nodes. These various link types may include, for example, an "AnchorsTo" type that states that one node is anchored to another node. Both nodes may use the Source Node or DestinationNode attributes as appropriate based on the parsing situation. Link types may also include type "Contains", describing that one node includes another node. In this relationship, a container node may be referred to as a SourceNode, while a container node may be referred to as a DestinationNode. The link types may further include a "PointTo" type, describing that one node points to another node. For this relationship, the pointing node may be referred to as a source node, while the pointing node may be referred to as a DestinationNode. In addition, link types may have a "PointsForm" type, describing that one node is pointed from another node. In this relationship, a node pointing from another node may be referred to as a source node, while a node pointing from another node may be referred to as a destination node.

The link types additionally describe the type "SpansHorizontally", which describes one node running the length of another node horizontally, and that one node executes the length of another node vertically, " May contain type SpansVertically. For these types, a node that normally covers the other node that is written last (strikeout, underline, margin bar) may be referred to as the SourceNode, while the spanned node is the destination. It may be referred to as a DestinationNode. Link types may also include a "Custom" type, which describes that a custom link type was used. If this value is used, the "CustomLinkType" attribute for the ContextLink object can provide more detail about the purpose of this link.

Thus, the ink analyzer API provides several functions and services for asynchronously analyzing electronic ink in a document, and then adjusts the results of the analysis process to the current state of the document as described above. In addition, it should be understood that the various classes described above may apply to various operating systems and operating environments, such as the Microsoft Windows operating environment, the Microsoft COM operating environment, the Unix or Linux operating environment, or any other suitable computer operating environment. In addition, it should be understood that an application programming interface in accordance with various implementations of the invention may omit one or more of the class components described above, or may include additional components to provide a desired service or functionality.

<Adjust>

As described above, various embodiments of the present invention allow the software application 1701 to continue to operate while the unanalyzed electronic ink in the document 1703 is analyzed by the background analysis process. Thus, the software application 1701 can change the document 1703 in a number of ways that would conflict with the results of the analysis process. For example, software application 1701 may enter new electronic ink data 1705 into document 1703, or delete existing electronic ink data 1705 from document 1703. In addition, the software application 1701 may edit the existing electronic ink data 1705 by moving the location of the existing electronic ink data 1705 or changing the attributes of the existing electronic ink data 1705. In addition, software application 1701 can add, delete, or change non-ink document elements 1707 in a manner that affects existing electronic ink data 1705. For example, the software application 1701 can delete the typed text indicated by the annotation in the electronic ink data 1705.

The software application 1701 may further "pin" the existing electronic ink data 1705 to prohibit changes by the analysis process. For example, if a user manually specifies a layout or classification for a group of ink strokes, the software application 1701 may indicate that the ink strokes remain in that classification, regardless of the results generated by the parsing process, or You can specify that it remains within a specific layout. Similarly, a user can specify a particular recognition result for a group of ink strokes, regardless of the results produced by the recognition process.

Various types of pinning can be used according to different implementations of the invention. For example, the ink analysis tool 1901 may allow the software application 1701 to "hard" fix the ink. In this arrangement, no ink strokes can be added to any leaf nodes below the fixed node, no strokes can be removed from any leaf nodes below the fixed node, and no addition of child nodes. Or delete is not allowed, and no re-parenting of any nodes below the fixed node is allowed. Alternatively or additionally, the ink analysis tool 1901 may allow for "hard" fixation with late strokes, which allows late strokes to be added under specified conditions, Prohibit from being removed from any leaf nodes below the node, prohibit the addition or removal of child nodes, and prohibit re-pairing of any nodes below the fixed node. In addition, the ink analysis tool 1901 may allow the software application 1701 to alternatively or additionally "soft" fix the ink. With this arrangement, no strokes can be removed from any leaf nodes below the fixed node, and there are rules specified to allow the addition of strokes (which allow it to be added internally for late strokes), Allow regrouping, addition and removal of child nodes. It should be noted that the pinning can be removed, and once pinning is removed, previously pinned nodes and their children can be considered "dirty" or require reparse.

In addition, ink processing according to various embodiments of the present invention may utilize multiple analysis processes, as discussed in detail above. Thus, while the second next analysis process is being executed, the results of the previous analysis process can change the electronic ink data 1705. Thus, since the results of the analysis process must be adjusted to the current state of the document 1703, only results valid for the current state of the document 1703 are applied to the analysis context object 1709. That is, the current analysis context object 1709 (and in some cases the results of the analysis process) excludes discrepancies or “collisions” between the results of the analysis process and the current state of the document 1703. To be changed.

Note that, in order for the results of the adjustment process to be valid, the state of the document 1703 should not be changed during the adjustment process. Thus, the adjustment process may be performed using the primary thread on which the software application 1701 is executed, and executing the adjustment process may temporarily stop the operation of the software application 1701. Alternatively, other techniques, such as data structure locking, may be used to ensure that the state of the document 1703 does not change during the reconciliation process. Therefore, in order to prevent the user from becoming unsatisfied with the performance of the software application 1701, it is preferable to perform the adjustment process as quickly as possible. Another consideration of the reconciliation process is the effect on the performance of the analysis process executed in a separate background analysis thread. If a discrepancy between the current analysis context object 1709 (ie, the analysis context object 1709 reflecting the current state of the document 1703) and the analysis results is defined too broadly, a large amount of electronic ink will be unnecessarily reanalyzed. Of course, other ways of protecting the integrity of the document during adjustment can be used without departing from the present invention.

In various embodiments of the present invention, analytical processing and coordination processing may cling to one or more of the following conventions to improve the efficiency and convenience of coordination processing. First, the analysis process can reuse as many nodes as possible for document elements in the document independent analysis context object 1903. In other words, conflicts should not be avoided by creating new unrelated nodes for document elements. Also, the adjustment process must respect the "pinning" specified by the software application 1701. The analysis process will typically conform to pinning specifications by the software application 1701, but the software application 1701 can freeze the electronic ink data 1705 while the analysis process is running. In addition, the reconciliation process must ensure that no empty nodes remain in the current analysis context object 1709 after the reconciliation process is completed. However, it should be understood that one or more of these conventions may be omitted or not obeyed according to alternative implementations of the invention. For example, some implementations of the invention may allow the analysis object 1709 to include empty nodes.

In addition to these conventions, the coordination process should typically follow the interface rules specified by the analysis context object 1709. For example, in some implementations of the invention, the analysis context 1709 may not allow a node for a document element to be deleted if it has no child nodes.

As mentioned above, a conflict occurs when a change to the document independent analysis context object 1903 which in some form opposes a change to the analysis context object 1709 made after the analysis process has been initiated. . Collisions can be divided into two types: mandatory and discretionary collisions.

A mandatory conflict occurs when it is impossible to apply a change made to the document independent analysis context object 1903 by the analysis process to the analysis context object 1709 for the current state of the document 1703. For example, software application 1701 "pins" or fixes a node in analysis context object 1709, and the analysis process changes the corresponding node in document independent analysis context object 1903. In this case, an essential conflict occurs. In addition, when the analysis process has made any kind of change to the node in the document independent analysis context object 1903, but the software application 1701 deletes the corresponding node from the analysis context object 1709, and the analysis processing is a document. An essential conflict will occur if the software application 1701 adds strokes or child nodes to a node in the analysis context object 1709 when deleting the corresponding node in the independent analysis context object 1903. In addition, an essential conflict is that the software application 1701 deletes a node in the analysis context object 1709 when the analysis process reorders or creates a link to the corresponding node in the document independent analysis context object 1903. Will happen if.

If the software application 1701 changes the value in the analysis context object 1709 associated with the value changed in the document independent analysis context object 1903 by the analysis process, an arbitrary conflict occurs, but the analysis context object 1709 Constraints, fixed constraints, and element reuse constraints inherent in the interface to may allow application of changes made by the analysis process to the analysis context object 1709. Arbitrary conflicts can be applied or avoided as a change to the analysis context object 1709. Also, the coordination process can simply ignore all types of arbitrary collisions together.

One graphical example of an arbitrary collision occurs when the original node of both the analysis context object 1709 and the document independent analysis context object 1903 has child nodes A and B for ink strokes A and B. . Software application 1701 may add a third child C node for ink stroke C within analysis context object 1709, while the analysis process may add a third for ink stroke D to document independent analysis context object 1903. Add child node D. In various embodiments of the present invention, the coordination process may add the child node D to the parent node in the analysis context object 1709 based on the analysis results. However, while this change to the parent node in the analysis context object 1709 is not prohibited, the properties of the parent node are changed in a way that may not be desired by the software application 1701. For example, the ink associated with the parent node may be considered to be subsequently analyzed and may not be reanalyzed to account for the effect of ink C on the group of ink strokes. Also, recognition results for the parent node can now refer to erroneous child nodes or strokes, which can never be corrected again.

Thus, various embodiments of the present invention will not apply changes to arbitrary conflicts when updating the analysis context object 1709 based on the results of the analysis process. Although this criterion may require additional processing to identify and block changes based on arbitrary conflicts, this criterion is relatively easy to implement and simple to maintain. Of course, still other examples of the invention may implement changes corresponding to arbitrary conflicts in accordance with other criteria. In particular, examples of such an alternative may implement changes from arbitrary conflicts that do not create permanent logical inconsistencies in the analysis context object 1709.

Note that collisions can have a transitive effect in that one collision can create another collision. For example, the analysis process may generate node L for the line and then generate node W for the word as a child node of node L. If the creation of node L was not applied to the analysis context object 1709 due to any kind of collision, then the creation of node W would be an essential conflict.

Various embodiments of the present invention may use a log-based approach to adjust the results of analytical processing to the current state of document 1703. In this log-based approach, document independent analysis context object 1903 includes a log of changes to document independent analysis context object 1903 made by an analysis process. The log may be in the form of a list of change records, for example. Each change is then made by the type of change that was made (e.g., by identifying the method that was called to change the document independent analysis context object 1903), (e.g., the document independent analysis context object 1903 from which the method was invoked). By identifying a node within the document element), and any information required to recreate arguments for method invocation. Advantageously, the document independent analysis context object 1903 is implemented by the ink analysis tool 1901 and the analysis processes, thus altering (even if the log may alternatively be exposed to the application 1701 if desired). The log may not be visible to the software application 1701.

In order to execute the adjustment process using the change log approach, the ink analysis tool 1901 examines each change record in the order of occurrence of changes. That is, the ink analysis tool 1901 can implement the processing shown in FIG. First, at step 2701, ink analysis tool 1901 attempts to access corresponding nodes in current analysis context object 1709 that are required to apply the change. This retrieval step may fail if the software application 1701 deletes one or more required nodes from the analysis context object 1709 and thus may lead to an essential conflict.

Next, at step 2703, the ink analysis tool 1901 determines whether the change produces the necessary conflict or the random conflict. The procedure for making this determination will be described in more detail below. In step 2705, the ink analysis tool 1901 makes a change or prohibits the change when generating an arbitrary collision or an essential collision prohibited by the criteria for the adjustment process. For example, if the change produces an arbitrary conflict or mandatory conflict that is prohibited by the criteria for the adjustment process, the ink analysis tool 1901 may analyze the analysis context object 1709 representing the corresponding area of the document 1703 in which the change was made. Changes can be blocked for nodes within On the other hand, once the change is applied, the ink analysis tool 1901 can call the appropriate method to make the desired change to the required nodes in the analysis context object 1709.

If the analysis process adds a new element node to the analysis context object 1709 but subsequently fails to move the stroke nodes to the new node, the analysis process also assumes that the stroke nodes have been successfully moved to the new nodes and deletes the element node. I will not try. Therefore, there will be an element node in document independent analysis context object 1903 corresponding to an empty node in current analysis context object 1709. Thus, once all changes to the analysis context object 1709 have been processed, in step 2707, the ink analysis tool 1901 is in the analysis context object 1709 corresponding to the nodes in the document independent analysis context object 1903. Review the document independent analysis context object to delete any "empty" nodes. This empty node deletion step is optional and can be omitted without departing from the present invention.

It should be noted that the document independent analysis context object 1903 should not include any empty nodes, and the adjustment process should prevent empty nodes from being created in the document independent analysis context object 1903. It should also be appreciated that this step, except in the absence of empty, will not attempt to delete the root node of the analysis context object 1709. Finally, at step 2709, the ink analysis tool 1901 will identify any conflicts between the document independent analysis context object 1901 and the analysis results of the current analysis context object 1709 for the software application 1701. The software application 1701 may include areas of the document 1703 that are affected by conflicts in the next analysis process.

Returning to the detection of conflicts in step 2703 now, once all nodes in analysis context object 1709 corresponding to changes in document independent analysis context object 1903 have been accessed, for each node in analysis context object 1709 All other possible mandatory collisions can be detected statically (or alternatively, the absence of mandatory collisions can be detected for each node). In particular, necessary conflicts can be detected based on rules specified by the interface to analysis context object 1709.

However, these changes are not enforced by the interface of the analysis context object 1709, but instead the changes made by the analysis process when both cases affect the same node in the analysis context object 1709. Since more favorable, based on discretionary selection of changes made by the software application 1701, arbitrary conflicts cannot be identified without additional status information about the document 1703. Instead, such conflicts are determined by comparing the current analysis context object 1709 with the original version of the document independent analysis context object 1903 to determine that the nodes within the analysis context object 1709 have been modified by the software application 1701. , Can be detected.

For example, the coordination criteria may define the addition or deletion of child stroke nodes to the parent node by the software application 1701 as being an arbitrary conflict against the parent node. To determine if such a change occurs, the ink analysis tool 1901 reviews the child stroke nodes that depend on the node in the current analysis context object 1709, and checks the corresponding node in the original version of the document independent analysis context object 1903. You can compare dependent child stroke nodes with their number and Guid (or other unique identifier). If the child stroke nodes match these embodiments (and any other desired embodiments), the ink analysis tool 1901 may apply the changes made by the analysis processes to the parent node in the analysis context object 1709. have.

Continuing with this example, the next change in the log entry may be made with another stroke node that is moved to depend on the parent node in the document independent analysis context object 1903. If there are no necessary conflicts from these changes, the ink analysis tool 1901 is allowed to make a change corresponding to the analysis context object 1709. However, when the ink analysis tool 1901 checks the original version of the document independent analysis context object 1903 to determine if any conflicts exist, then the parent node in the current analysis context object 1709 is the document independent analysis context object ( It will be concluded that it has more than one dependency stroke node than the parent node in the original version of 1903, thus it will make a false decision that a mandatory conflict exists (i.e., determine that a collision exists based on changes made). will be).

Thus, the log-based reconciliation process must exclude previous changes made by the reconciliation from the comparison between the original version of the document independent analysis context object 1903 and the current analysis context object 1709. In various embodiments of the present invention, this exclusion may include all nodes in the current analysis context object 1709 prior to examining any list entries and maintaining a list of conflicting nodes. By comparing with nodes corresponding to them in the original version of 1903.

Alternatively or additionally, various embodiments of the present invention provide a cache of elements of the current analysis context object 1709 that correspond to the elements of the document independent analysis context object when the corresponding elements are retrieved. You can keep the analysis results used every time. There is no preliminary delivery, but the cache is populated when the corresponding element is first determined to be free of arbitrary collisions. Optionally, the cache may record which elements have arbitrary collisions as a result. This approach has better performance than the discretionary-collision-finding-pass approach when the change log is short, but the document independent analysis context object 1903 with analysis results includes many nodes. If you do, you may have worse performance. It is also not more complicated than other approaches.

It should also be noted that the parent context nodes of the elements in the cache should be cached. In particular, a change to a child context node will be considered a change to the attributes of its parent context node. Thus, also for changes made by the software application 1701 at this point, the parent context node of the context node stored in the cache needs to be evaluated. Caching of the parent nodes for each context node in the cache is performed until the root node is delivered or until changes are detected, up to the end of the tree (e.g., from grandparent to grandparent to grandparent). (great-grandparent) nodes need to be repeated. However, in many situations, the tree will typically not be very deep, so this iteration is only done for a few nodes.

Also in various examples of the invention, the original version of the document independent analysis context object 1903 is used only for the purpose of detecting arbitrary conflicts. If detection of arbitrary conflicts is not essential in the examples of the present invention, both the original version of the document independent analysis context object 1903 and the corresponding element cache may be removed. This "log-based" approach requires less searching to determine what changes have been made by the analysis process and because the operations on the document independent analysis context 1903 are handled separately, the " It may be easier to maintain than a "comparison-based" approach, so adding a new type of operation does not easily affect other operations.

Instead of a "log-based" technique for adjusting analysis results to the current state of document 1703, various embodiments of the present invention may use a "comparison-based" approach for adjustment. The main distinguishing feature of the comparison-based approach is that no log of changes made to the document independent analysis context object 1903 is maintained, so this technique allows document independent analysis context objects containing analysis results to be document independent. It does not collect information about what the analysis process did, except by comparing it with the original version of the analysis context object 1903. Therefore, in this approach, the original version of the document independent analysis context object 1903 is always required, regardless of any judgment on arbitrary conflicts.

Using this approach, the ink analysis tool 1901 first builds a stroke map. That is, for every ink leaf node in the original version of the document independent analysis context object 1903, if there is a corresponding node in the current analysis context object 1709, the leaf node is a hash table (or other appropriate data). Structure). Thus, the hash table maps the stroke GUIDs (or other single node identifiers) in the original version of the document independent analysis context object 1903 to leaf node references in the current analysis context object 1709.

Next, the ink analysis tool 1901 identifies all nodes that the software application 1701 changes in the analysis context object 1709. This is done using techniques for determining the arbitrary conflicts described above. For each node in the original version of the document independent analysis context object 1903, there is a corresponding node in the current analysis context object 1709 and it is in some important way a node with the node in the original version of the document independent analysis context object 1903. If different, the ink analysis tool 1901 determines that this node has been modified by the application and is a potential conflict. References to such nodes are included in a first list corresponding to analysis result nodes that should not be added to analysis context object 1709 (hereafter referred to as resultsNodesNotToAdd list) and in a second list of nodes that generate arbitrary conflicts. Stored.

For each node in the original version of the document independent analysis context object 1903, the ink analysis tool 1901 may select the document independent analysis context object 1903 with the analysis results and the corresponding nodes in the current analysis context object 1709. Find. For each node in analysis context object 1709, ink analysis tool 1901 determines whether software application 1701 has deleted the node. Such nodes exist in the original version of the document independent analysis context object 1903, but there are no corresponding nodes in the analysis context object 1709. The corresponding node in the document independent analysis context object 1903 with analysis results, if present, is added to the resultsNodesNotToAdd list.

Next, the ink analysis tool 1901 determines whether the node has been deleted from the document independent analysis context object 1903. This is detected when there is a corresponding node in analysis context object 1709 and there is no corresponding node in document independent analysis context object 1903 with analysis results although it has not changed. Nodes deleted by the analysis process are stored in the list of deleted nodes. Such a node is not currently added to the analysis context object 1709, but is not removed from its resultsNodesNotToAdd list because there is no corresponding node in the document independent analysis context object 1903 with analysis results.

If the software application 1701 did not delete or change the node, and the analysis process did not delete the node, the ink analysis tool 1901 may determine whether the analysis process changed the node. The same techniques described above for use to detect changes made by the software application 1701 can also be used to detect changes made by the analysis process. Nodes changed by the analysis process are added to the resultsNodesNotToAdd list, and the changes are propagated by invoking the nodes in the analysis context object 1709 in an appropriate manner. If the node has not been changed or deleted by either the software application 1701 and the analysis process, nothing has happened to that node, and the node resides in the document independent analysis context object 1903 with the analysis results and the analysis context object ( Since it is not added to 1709, the ink analysis tool 1901 adds the node to the resultsNodesNotToAdd list.

Next, for each node in document independent analysis context object 1903 with analysis results, if the node is not in the resultsNodesNotToAdd list, ink analysis tool 1901 analyzes by creating a corresponding node in analysis context object 1709. Any changes to the node by the process can be propagated. Traversal of nodes in the analysis results can be performed using top-down pre-order search, so that parent nodes are guaranteed to be created before child nodes.

When creating a new leaf node in the analysis context object 1709, any ink strokes associated with the node included in the analysis results may also be moved from other nodes in the analysis context object 1709 to the new nodes. The stroke map described above can then be used to identify source elements for moving the strokes. If there is no source element (creating a mandatory collision), or if the source element is included in the arbitrary collision list described above, the ink strokes are not moved. To store unnecessary calls to analysis context object 1709, this ink analysis tool 1901 invokes a method to create a child context node when there is no at least one non-colliding stroke source node. Can be avoided. The same stroke source check is also performed at non-ink leaf nodes, so if the component words of at least one line do not contain, for example, at least one successfully moved stroke, a node for the line is created. It doesn't work.

Subsequently, other information is also applied to the newly created node, such as the recognition grid. The recognition grid applies even when arbitrary strokes cannot move while creating an applied (i.e. non-blocked) random collision resulting in temporary logical inconsistency. However, because the failed stroke movements are reflected in the designation of the portion of the analysis context object 1709 that corresponds to the area of the document in which changes have been blocked, the inconsistency because the software application 1701 is notified to reanalyze the blocked area. Is temporary.

For each node to be deleted from the analysis context object 1709, the ink analysis tool 1901 list checks to see if the node on which all other edits and additions to the analysis context 1709 have been performed can actually be deleted. do. Nodes that must be deleted and all child nodes and strokes are actually empty are deleted from the analysis context object 1709. Once such a node is deleted, all ancestor nodes (up to the root node but not the root node) are deleted sequentially until they encounter an ancestor with other remaining children.

Finally, the ink analysis tool 1901 can analyze the document independent analysis context object 1903 with the analysis context object 1709 and analysis results to detect and propagate changes to the analysis context object 1709 by the analysis process. The overall comparison of the nodes that determine the order between the two is performed. However, it should be noted that since this comparison ignores the original version of the document independent analysis context object 1903, there is no way to detect changes to the order determined by the software application 1701. Next, for each container node in the analysis results having the corresponding node in analysis context object 1709, the order of the children of container nodes is compared. If both container nodes have exactly the same set of children, the order of the analysis results is a list of containers in the analysis context object 1709 where the node does not have the same GUID (or other identifier type) as the child node of the analysis results. It is propagated to the analysis context object 1709 by looping through the children of the container in the analysis results until found at the same location within. If such a mismatch is found, the corresponding node must also be found further down the child list in the analysis context object 1709. The ink analysis tool 1901 then retrieves the remainder of the child list in the analysis context object 1709 for the node and calls a method to move the child node to modify the command for the node in the analysis context 1709. . The ink analysis tool 1901 then loops continuously through the nodes in the analysis results.

Of course, the analysis list or software application 1701 may have nodes deleted or inserted in any list, so the assumptions about the same lists do not hold true. A "child list mapper" object is used to simulate the assumption of keeping true by presenting "pruned" lists that only contain elements that are kept in common. The above procedure is executed on the mapped child lists, and moves are translated into actual indices and calls by the list mapper object.

Therefore, the adjustment techniques according to the various embodiments of the present invention described above allow the results of the analysis process to be applied to the current document elements in the document even if the contents of the document are changed after the analysis process is started. Therefore, these adjustment techniques allow the electronic ink in the document to be analyzed asynchronously with the operation of the software application hosting the document. Such coordination techniques may also be used by a variety of different software applications, including software applications multithreaded with existing proprietary locking or other synchronization strategies.

<Event Drive System>

As noted above, various embodiments of the present invention create a "snapshot" of the document state by copying a document independent analysis context object, followed by a document independent analysis while the software application hosting the document continues to operate. Parse context objects asynchronously. Alternatively, various embodiments of the present invention may precede the use of a document independent analysis context object for asynchronous ink analysis. Instead, these examples of the invention may use a sealed component for storing all the ink and semantic data for the document. In particular, these examples of the present invention may include documents: ink events such as adding, deleting or changing strokes; And two types of changes that can be made to structural events such as grouping strokes into words, adding semantic nodes, or associating text recognition results to strokes. Every event contains all the data necessary to completely describe the event to an external listener. For example, the "Add Stroke" event will contain all the stroke data. As such "rich" events, the listener can maintain an accurate copy of the ink object by applying the events in the order received.

Figure 28 shows one example of how this arrangement may be used, in accordance with various embodiments of the present invention. As shown in this figure, the application 2801 uses an ink analysis tool 2803. Ink analysis tool 2803 maintains ink data and document structure 2807 (such as, for example, a tree structure) for application 2801. The application also utilizes an event queue 2809, parser processing 2811 and recognition processing 2817. The parser process 2811 holds a clone of the ink data 2805 and a clone 2815 of the document structure 2807. Similarly, the recognition process 2817 holds a clone 2819 of the ink data 2805 and a clone 2821 of the document structure 2807.

The application provides the ink data to the ink analysis tool 2803 through the method 2823 when generating the ink data. In response, the ink analysis tool 2803 can generate an event corresponding to a change in the ink data, add a tag to every event, and listen to the desired analysis with components that listen for events with specific tags. Define a process or processes. For example, all user changes to the ink can be marked with the tag "UserChange", and an event 2825 with this tag is sent to the event queue 2809 synchronously in response to the change. At some point in the future, parser processing 2811 and recognition processing 2817 will retrieve tagged events from event queue 2809.

The first analysis process (eg, parsing process 2811) will then listen for all events issued by the ink analysis tool 2803 and respond to those with the tag "UserChange". The first analysis process then applies the changes specified in the event to an internal copy of the ink data 2813 and the document structure 2815, analyzes the data described in that event, and then generates the events ( 2827) and tag it with the tag "Parser1Change". These events 2827 will be returned to the event queue 2809 and relayed by the ink analysis tool 2803. The second analysis process (eg, handwriting recognition process 2817) will then listen for events tagged with the "Parser1Change" tag. In response, this will apply the changes specified by the event to an internal copy of ink data 2819 and document structure 2821 and analyze the data described in the events in response. The second analyzer will then generate and tag the event 2829 with analysis results with the tag "HandwritingRecognitionChange". Ink analysis tool 2803 sends events 2831 to application 2801 to indicate changes in document structure 2807 when receiving events from parser processing 2811 and recognition processing 2817. .

These embodiments of the present invention may also support each analysis process by following events with one or more events, and tag those events from changes with other tags based on internal processing. Thus, these alternative implementations of the present invention may also consider the analysis of electronic ink in a document asynchronously from the operation of the software application hosting the document.

Conclusion

Although the present invention has been described with respect to specific examples which now include the preferred modes of carrying out the invention, those skilled in the art will appreciate that systems and techniques described above that fall within the spirit and scope of the invention as set forth in the appended claims. It will be understood that there are various changes and exchanges.

Claims (65)

Electronic ink tin 306, 512, 752, 802, 902, 1002, 1102, 1104, 1106, 1202 of documents 300, 500, 750, 800, 900, 1000, 1100, 1200, 1300, 1400, 1500, 1703 1302, 1304, 1306, 1308, 1310, 1402, 1404, 1406, 1408, 1410, 1502, 1705. A method executed on a computer for asynchronous reception and processing of a document, wherein the document is an electronic document. Generating a first analysis context object 404, 1709, the first analysis context object providing a translation layer for the document model 402 of the current state of the relationship of the elements in the electronic document; Tree data structures for storing them in a hierarchical relationship (502, 600, 610, 630, 630a, 640, 650, 654, 680, 710, 760, 810, 910, 1110, 1210, 1320, 1420, 1510, 1530, 1810); Starting a first thread, the first thread updating the first analysis context object based on a user interaction with the electronic document, the user interaction comprising an electronic ink annotation; If an event requires the analysis of new data in the electronic document, Suspending execution of the first thread to prevent changes to the first analysis context object; Starting a second thread, the second thread creating a second analysis context object 1903 corresponding to the portion of the first analysis context object, the portion corresponding to a designated area of the electronic document; Upon completion of generation of the second analysis context object, Suspending execution of the second thread; Resuming the first thread; Executing a first analysis on the second analysis context object to generate a third analysis context object 2101 from the second analysis context object, wherein the first analysis comprises: parsing the new data; and Modifying the second analysis context object based on new data, wherein the third analysis context object further includes classification information for the new data; Upon completion of the first analysis, Suspending execution of the first thread to prevent any changes to the first analysis context object; Starting a third thread, the third thread adjusting the third analysis context object to the first analysis context object to produce first adjusted analysis results 2401; Upon completion of the adjustment of the first analysis context object and the third analysis context object, Updating the first analysis context object with the first adjusted analysis results; Suspending execution of the third thread; And Resuming the first thread A computer executable method for asynchronously receiving and processing electronic ink annotation of a document comprising a. The method of claim 1, The first analysis context node may include a paragraph node 614, 644, 654, 672, 764, 1018, 1330, 1424a, 1424b, 1424c, 1528a, 1528b, 1528c, 1528d, and a line node. (616, 646, 656, 674, 766, 1020, 1022, 1332, 1426a, 1426b, 1426c), group of word nodes (648, 658, 676, 768a-768e) and drawing nodes A computer executable method comprising members selected from. The method of claim 1, The first analysis context object is a group node (634a, 668), paragraph node (614, 644, 654, 672, 764, 1018, 1330, 1424a, 1424b, 1424c, 1528a, 1528b, 1528c, 1528d), line node (616, 646, 656, 674, 766, 1020, 1022, 1332, 1426a, 1426b, 1426c), ink word nodes (618, 660, 1024, 1026, 1028, 1132, 1334, 1428a, 1428b, 1428c), electronic Picture node, ink picture node 636, 636a, 770, 812, 912, 1212, 1324, 1342, 1432a, 1432b, 1432a, 1432b, list node 684, list item node 686, Electronic bullet node, ink bullet node 688, electronic text word nodes 648, 658, 676, 768a-768e, image nodes 670, 698, 720, table nodes 714, 1514, And member selected from the group of row nodes (1516, 1518), and cell nodes (1520, 1522, 1524, 1526). The method of claim 3, The second analysis context object includes unclassified ink nodes 604, 1825, 1827, group nodes 634a, 668, shorting nodes 614, 644, 654, 672, 764, 1018, 1330, 1424a, 1424b, 1424c, 1528a, 1528b, 1528c, 1528d), line nodes 616, 646, 656, 674, 766, 1020, 1022, 1332, 1426a, 1426b, 1426c, ink word nodes 618, 660, 1024, 1026 , 1028, 1132, 1334, 1428a, 1428b, 1428c), ink drawing node 636, 636a, 770, 812, 912, 1212, 1324, 1342, 1432a, 1432b, 1432a, 1432b), list node 684, list Computer executable, selected from the group of item nodes 686, ink bullet nodes 688, table nodes 714, 1514, row nodes 1516, 1518, and cell nodes 1520, 1522, 1524, 1526. Way. The method of claim 1, Wherein the portion comprises at least one of electronic text, an image, a table, a list, a graph, a spreadsheet, a chart, or a picture. The method of claim 1, The new data is an electronic ink annotation, the annotation comprising at least one unclassified ink node (1823, 1825, 1827). The method of claim 6, Rendering the portion and the electronic ink tin, wherein the electronic ink tin is positioned in a first position relative to the portion; Altering data associated with the portion to change a location associated with the first analysis context object to a second location; Rendering the portion and the electronic ink annotation with the modified data, wherein the electronic ink annotation is rendered at a third position relative to the portion based at least in part on the second position of the first analysis context object. Further comprising, a computer executable method. The method of claim 6, The data associated with the first analysis context object and the second analysis context object enable the electronic document to be rendered such that the electronic ink annotation includes the portion of the electronic document. The method of claim 6, Data associated with the first analysis context object and the second analysis context object enables the electronic document to be rendered such that the electronic ink annotation underlines the portion of the electronic document. The method of claim 6, The data associated with the first analysis context object and the second analysis context object enables the electronic document to be rendered such that the electronic ink annotation marks a strike out in the portion of the electronic document. , Computer executable method. The method of claim 6, The data associated with the first analysis context object and the second analysis context node may be configured such that the electronic document is pointed so that the first area of the electronic ink annotation points between the second area of the electronic ink annotation and the portion of the electronic document. A computer executable method that enables rendering. The method of claim 6, Wherein the portion corresponds to one or more words of typed text in the electronic document and the annotation is an electronic ink comment of the one or more words of typed text. The method of claim 6, Wherein the portion corresponds to an electronic ink picture in the electronic document and the annotation is an electronic ink comment of the electronic ink picture. The method of claim 1, And the first analysis context object and the second analysis context object share at least one common parent node. A computer readable storage medium storing computer executable instructions for performing the method of claim 1 when executed by a computer. The method of claim 1, After execution of the first thread is suspended, initiating a caching thread to receive changes to the electronic document based on harvest user interactions; Suspending execution of the caching thread after execution of the second thread is suspended; After the execution of the first thread is suspended, if the first analysis is complete, resuming the caching thread; And Suspending execution of the caching thread after execution of the third thread is suspended. Electronic ink tin 306, 512, 752, 802, 902, 1002, 1102, 1104, 1106, 1202 of documents 300, 500, 750, 800, 900, 1000, 1100, 1200, 1300, 1400, 1500, 1703 A system for asynchronous reception and processing of 1302, 1304, 1306, 1308, 1310, 1402, 1404, 1406, 1408, 1410, 1502, 1705, Input devices (165, 166) for receiving electronic ink input data in the document, wherein the document is an electronic document; And A processor 110, The processor comprising: Create a first analysis context object 404, 1709-the first analysis context object providing a translation layer for the document model 402 of the current state of the relationship of the elements in the electronic document and hierarchical document elements Tree data structures (502, 600, 610, 630, 630a, 640, 650, 654, 680, 710, 760, 810, 910, 1110, 1210, 1320, 1420, 1510, 1530, 1810) Includes-; Initiate a first thread, the first thread to update the first analysis context object based on a user interaction with the electronic document, the user interaction comprising an electronic ink annotation; If an event requires the analysis of new data in the electronic document, Suspend execution of the first thread to prevent changes to the first analysis context object; Initiate a second thread, the second thread creating a second analysis context object 1903 corresponding to the portion of the first analysis context object, the portion corresponding to the designated area of the electronic document; Upon completion of generation of the second analysis context object, Suspend execution of the second thread; Resume the first thread; Perform a first analysis on the second analysis context object to generate a third analysis context object 2101 from the second analysis context object, wherein the third analysis context object parses the new data and the new analysis context object; Generated by modifying the second analysis context object based on data; Upon completion of the first analysis, Suspend execution of the first thread to prevent any changes to the first analysis context object; Initiate a third thread, the third thread adjusting the third analysis context object to the first analysis context object to produce a first adjusted analysis result 2401; Upon completion of the adjustment of the first analysis context object and the third analysis context object, Update the first analysis context object with the first adjusted analysis result; Suspend execution of the third thread; And And programmed and adapted to resume the first thread. The method of claim 17, The first analysis context object may include a paragraph node 614, 644, 654, 672, 764, 1018, 1330, 1424a, 1424b, 1424c, 1528a, 1528b, 1528c, 1528d, and a line node. (616, 646, 656, 674, 766, 1020, 1022, 1332, 1426a, 1426b, 1426c), group of word nodes (648, 658, 676, 768a-768e) and drawing nodes A system comprising members selected from. The method of claim 17, The first analysis context object is a group node (634a, 668), paragraph node (614, 644, 654, 672, 764, 1018, 1330, 1424a, 1424b, 1424c, 1528a, 1528b, 1528c, 1528d), line node (616, 646, 656, 674, 766, 1020, 1022, 1332, 1426a, 1426b, 1426c), ink word nodes (618, 660, 1024, 1026, 1028, 1132, 1334, 1428a, 1428b, 1428c), electronic Picture node, ink picture node 636, 636a, 770, 812, 912, 1212, 1324, 1342, 1432a, 1432b, 1432a, 1432b, list node 684, list item node 686, Electronic bullet node, ink bullet node 688, electronic text word nodes 648, 658, 676, 768a-768e, image nodes 670, 698, 720, table nodes 714, 1514, A member selected from the group of row nodes (1516, 1518), and cell nodes (1520, 1522, 1524, 1526). The method of claim 19, The second analysis context object includes unclassified ink nodes 604, 1825, 1827, group nodes 634a, 668, shorting nodes 614, 644, 654, 672, 764, 1018, 1330, 1424a, 1424b, 1424c, 1528a, 1528b, 1528c, 1528d), line nodes 616, 646, 656, 674, 766, 1020, 1022, 1332, 1426a, 1426b, 1426c, ink word nodes 618, 660, 1024, 1026 , 1028, 1132, 1334, 1428a, 1428b, 1428c), ink drawing node 636, 636a, 770, 812, 912, 1212, 1324, 1342, 1432a, 1432b, 1432a, 1432b), list node 684, list The system is selected from the group of item nodes (686), ink bullet nodes (688), table nodes (714, 1514), row nodes (1516, 1518), and cell nodes (1520, 1522, 1524, 1526). The method of claim 17, The portion includes at least one of electronic text, an image, a table, a list, a graph, a spreadsheet, a chart, or a picture. The method of claim 17, The new data includes electronic ink tin, prior to parsing the electronic ink tin, the tin including at least one unclassified ink node. The method of claim 22, The processor comprising: Render the portion and the electronic ink annotation, wherein the annotation is located in a first position relative to the portion; Receive an input indicating a change in data associated with the portion to change a place associated with the first analysis context object to a second location; And further programmed and adapted to render the portion and the electronic ink annotation with the modified data, the annotation being rendered at a third position for the portion based at least in part on the second position of the first analysis context object. System. The method of claim 22, The data associated with the first analysis context object and the second analysis context object enable the electronic document to be rendered such that the electronic ink annotation includes the portion of the electronic document. The method of claim 22, The data associated with the first analysis context object and the second analysis context object enable the electronic document to be rendered such that the electronic ink annotation underlines the portion of the electronic document. The method of claim 22, Data associated with the first analysis context object and the second analysis context object enables the electronic document to be rendered such that the electronic ink annotation marks a strike out in the portion of the electronic document. , system. The method of claim 22, Data associated with the first analysis context object and the second analysis context object is rendered such that the electronic document is rendered such that the first area of the electronic ink annotation is between the second area of the electronic ink annotation and a portion of the electronic document. System, which makes it possible to become. The method of claim 22, Wherein the portion corresponds to one or more words of typed text in the electronic document and the annotation is an electronic ink comment of the one or more words of typed text. The method of claim 22, Wherein the portion corresponds to an electronic ink picture in the electronic document and the annotation is an electronic ink comment of the electronic ink picture. The method of claim 17, And the first analysis context object and the second analysis context object share at least one common parent node. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images