KR20070010094A - Electronic ink processing - Google Patents

Abstract

A method of analyzing electronic ink (Fig. 5, 502), in which document data for a document containing electronic ink content is received from a software application running on a first processing thread. The first processing thread is employed to provide the document data (Fig. 5, 500) to an electronic ink analysis process for analyzing on a second processing thread. Control of the first processing thread is then returned to the software application. After the results of the analysis are received, the results are reconciled with the current document data for the document. ® KIPO & WIPO 2007

Description

Electronic ink processing {ELECTRONIC INK PROCESSING}

The present invention relates to the treatment of electronic ink. In particular, various embodiments of the present invention are applicable to the analysis of electronic ink, including layout analysis, classification, and recognition of electronic ink. Additional embodiments of the present invention relate to the use of layout analyzed, classified and recognized electronic ink, for example in providing rich and manageable annotations in an electronic ink document.

As the role of computers in society increases, several different techniques for entering data into computers have been developed. One notably useful technique for entering data is by handwriting. By writing with a stylus or other object on the digitizer to produce "electronic ink", computer users can lessen the volume and inconvenience associated with the keyboard. Handwriting input can be used, for example, in the attending physicians, the architects of the building site, the deliverers delivering the parcel, the warehouse workers walking around the warehouse, and any situation where the use of the keyboard is difficult or inconvenient. Handwriting input is more convenient than a keyboard in many situations, but text written in electronic ink cannot typically be handled directly by most software applications. Instead, text written in electronic ink must be analyzed to convert it to another form of handwriting, such as ASCII characters. This analysis involves handwriting recognition processing, which recognizes characters based on the various relationships between the individual electronic ink strokes that make up a word of electronic ink.

Handwriting recognition algorithms have been greatly improved in recent years, but can be less accurate if the electronic ink is written at an angle. Likewise, if separate groups of ink strokes cannot be easily distinguished, such as when two words are written together close together, many recognition algorithms cannot correctly recognize the electronic ink. Some recognition algorithms may also misrecognize electronic ink as text, in fact when electronic ink is intended for drawing. For example, the user may type by typing an underline, highlight, or circle or cross-marked electronic ink stroke on a portion of the typewritten text. Commented text. The handwriting recognition algorithm may then mistakenly recognize these annotation strokes as a dash, a zero, or the letter "O".

The accuracy of many recognition algorithms can be significantly improved by "parsing" the electronic ink (eg, by analyzing the layout of the electronic ink and / or by "classifying" the electronic ink) before using the handwriting recognition algorithm. Can be. The electronic ink stroke typically determines whether the classification process is typically part of a picture (ie, picture ink stroke), otherwise it is part of handwritten text (ie, text ink stroke). Classification algorithms for identifying other stroke types are also possible. The layout analysis process typically groups the electronic ink strokes into meaningful combinations such as words, lines and paragraphs. Therefore, the layout analysis and classification processes include which strokes in the collection of electronic ink belong to one word, which words of the electronic ink are associated with one line of text written in electronic ink, and which of the text written in electronic ink. Can be used to identify if the lines are associated with a short.

Although the layout of analyzing and classifying inks can greatly improve the perception of electronic ink, many software application developers are unaware of the importance of these activities before recognizing electronic ink. Until recently, layout and classification algorithms were not readily available to use existing software applications. For example, to store, display, and manipulate electronic ink, Microsoft? Microsoft with Windows Journal software application? The Windows XP Tablet PC Edition version 2002 operating system was commonly sold. Until recently, Microsoft? Although the Windows Journal software application used an internal parser, it could not access other software applications run by the operating system.

Although parsing processing from Windows Journal software applications is now accessible separately from other software applications, the use of such parsers is not well known, and such parsers are easily used with many software applications where the user wants to enter handwriting input. Can be. Moreover, even if the software application developer creates a parser to use the desired software application (which can be difficult and time-consuming in itself), the execution of the parsing process can be very time-consuming. For example, parsing only a few strokes of electronic ink using a relatively fast microprocessor may take seconds or even minutes. If a software application needs to stop working until the parsing process is complete, the software application will be too slow for most users to use practically.

Accordingly, electronic ink processing techniques that can be used by various software applications, for example, need to analyze the layout of the electronic ink and classify and / or recognize the electronic ink. In addition, there is a need for electronic ink processing techniques capable of processing electronic ink, but there is also a need to allow software applications using these technologies to accept new electronic ink input without invalidating the results of the ink processing.

Brief overview of the invention

Advantageously, various embodiments of the present invention provide electronic ink processing techniques that can be used by various software applications to process electronic ink. In addition, these electronic ink processing techniques allow the electronic ink to be processed asynchronously with respect to the operation of the software implementing the techniques, so that the electronic ink can be processed without stopping or significantly delaying the operation of the software application. The software application can continue to accept new electronic ink input even while the previous electronic ink input is being processed.

As various embodiments of the present invention, elements of a file or document may be described based on their spatial location relative to each other. For example, both the electronic ink stroke and the typed text can be described by the same spatial coordinate system. Using spatial information to describe the elements of a document, the software application managing the document can maintain a data structure describing the relationships between the document elements. In particular, the software application may describe a class of various document elements and maintain a data structure that defines combinations between the various document elements. Such combinations may be defined, for example, as information used to link electronic ink stroke data or combinations thereof to other elements (such as words, lines, paragraphs, drawings, table cells, etc.) in an electronic document. .

By describing document elements in a file or document data structure based on spatial location, document elements for various file types can use common techniques to identify and manipulate those document elements. In particular, various software applications can describe document elements in a document based on their spatial location and use this spatial location reference to use common electronic ink analysis methods. In addition, by designating a specific area of the document for analysis, each software application can limit the analysis process to only the desired elements in the document.

In order to analyze a new electronic ink input to a document in accordance with various embodiments of the present invention, the software application managing the document changes the data structure associated with the document to include the new ink to be analyzed. The software application then provides this data structure (or its related portions) to the ink analysis tool, which copies some or all of the data structure for analysis (and independently of the document data structure of the application). Activate a copy of the data). The ink analysis tool delivers its copy to an analysis process, such as a parsing process (eg, layout analysis process and / or classification process). The software application may resume normal operation including receiving new electronic ink input and / or other data while the ink analysis processing (s) is being performed. In addition to receiving a new electronic ink, any “other data” includes content such as an application, for example, size, location or existing ink, text, images, graphics, tables, flowcharts, diagrams, and the like. Data to make; Data to add additional text, images, graphics, tables, flowcharts, diagrams, and the like; It may be received by data that deletes existing text, images, graphics, tables, flowcharts, diagrams, and the like. After all desired analysis processes have been completed, the analysis results are returned to the ink analysis tool.

Therefore, various examples of systems and methods in accordance with the present invention allow ink analysis processes to be executed asynchronously from the operation of a software application using the ink analysis process. This asynchronous operation allows the user to continue using software applications that are not delayed by the analysis process. In addition, this allows multiple analysis processes to be executed simultaneously.

In response to receiving the analysis results, the ink analysis tool obtains a current version of the data structure of the electronic document (which may include new and / or changed data entered while the analysis processes are performed) from the software application. Adjust the analysis results of the current version of the data structure. By adjusting the analysis results of the current version of the data structure, various embodiments of the present invention can avoid more complex techniques, such as "locking", to asynchronously access the data being used by the software application. Instead, the reconciliation can be called by any software application without having to prepare for complex internal locking.

After adjusting the analysis results of the current version of the data structure, the ink analysis tool can provide a copy of the adjusted analysis results to another analysis process, such as a handwriting recognition process. Again, the software application may resume normal operation, including receiving new electronic ink input and / or other data while the second ink analysis processing (s) is being performed. After all of the desired second analysis processes are completed, the results of the second analysis process are returned to the ink analysis tool. The ink analysis tool then obtains the current version of the data structure from the software application (which may include new and / or changed data again) and adjusts the second analysis results of the current version of the data structure. The ink analysis tool then updates the data structure using the adjusted results of the second analysis process. Of course, any number of ink analysis procedures and / or steps may be used without departing from the present invention.

The use of spatial information to associate or link electronic ink data with the various ink analysis processes and other features of the electronic document described above can be used to provide rich, easy to handle, plain ink annotations within an electronic document. For example, embodiments of the present invention can be used to provide electronic ink annotations that move enormously and / or otherwise change based on changes the underlying document elements are intended to be annotated. Users typically can, for example, circle, underline, highlight, or strikeout the words; Can be indicated in margin; Annotate documents in many different ways, such as drawing with annotations placed in margins with arrows or other pointers. In addition, users can annotate with a wide variety of different document types, including, for example, text, spreadsheets, drawings, slideshows, tables, charts, graphs, flowcharts, and the like.

Smooth integration of electronic ink annotations into an electronic document requires the annotations to work properly when the underlying electronic document is changed for some reason. For example, if a user marks a word in an electronic document (as a comment) and then adds text somewhere in the document before that word, it may cause the circle marked word to move. In this case, the circle comment should move and stop the word. As another example, if a user adds or removes characters from the circle-marked word, or otherwise changes its size in any way, the circle-marked comment must expand or contract to accommodate the new size of the word. do. Preferably, if the underlying electronic document reflows the positions of elements (paragraphs, drawings, columns, etc.) with respect to each other and / or updates the page for some reason, the electronic ink annotations also have Reposition or appropriately position for text or other information. If the annotations do not work this way on the underlying electronic document text, then it becomes impractical to edit or share the vivid electronic documents after the annotation, and the parer printout should be the easiest and most useful way to annotate. will be.

Determining how to properly change annotations will depend on various factors regarding annotations and electronic documents. For example, appropriately changing tin in a reflow situation may include, for example, (a) identifying the electronic ink as tin; (b) identify the type of electronic ink tin; (c) may rely on the computing system's (parser's) ability to identify the relationship of the electronic ink to a particular element in the underlying electronic document. It may be possible to query the user for some or all of these types of information, but such systems result in a much more cumbersome user experience than annotating on paper. Thus, according to various embodiments of the present invention, the information can be deduced from the content of the underlying document, including (1) the electronic ink itself and (2) the spatial location of the various elements within the document relative to the electronic ink. have.

Since the determination of the meaning of ink annotations can be complex and difficult, it is very impractical to expect each application of a document that a user wishes to annotate to implement annotation identification logic separately. Therefore, it is desirable to provide a reusable component to assist in this annotation function that each application can easily configure. For example, because the operating system of a pen-based computing system provides components for the collection and rendering of electronic ink, the meaning of ink annotations that can work very easily when filled with ink It would be desirable for the operating system to provide a component that determines (also referred to as an "annotation parser").

However, identifying the content of the underlying electronic document poses substantial difficulties when providing "smart" annotation capabilities. For example, many different applications have very different ways of storing, managing, and reflowing documents. The present invention provides a reusable annotation parser that can determine the relationship between various different types of electronic documents and possible ink annotations. In particular, according to some embodiments of the present invention, relevant information about an ink being parsed into a mechanism (eg, information about an electronic document in a spatial area corresponding to the ink annotation to provide an "context" to the ink annotation). An annotation parser is provided that calls back to the application to provide. The mechanism is easy enough to practice to integrate into any document-based application and is efficient enough to work with very large documents (which can be processed in parts such as pages).

Therefore, ink processing techniques in accordance with various embodiments of the present invention allow various software applications to perform multiple processes for electronic ink through an ink analysis tool. In addition, a software application using these techniques can continue normal operation during an analysis process that includes receiving a new electronic ink input without necessarily invalidating the results of the analysis processes.

The above and other objects, features and advantages of the present invention will be readily apparent and fully understood from the following detailed description set forth in conjunction with the accompanying drawings.

1 is a schematic diagram of a general-purpose digital computing environment in which some embodiments of the present invention may be implemented.

2 is a schematic diagram of a pen-based personal computing (PC) environment in which some embodiments of the present invention may be implemented.

3 and 4 illustrate various features of exemplary implementations of embodiments of the present invention with regard to electronic document annotations.

5 illustrates general features of ink parsing of an electronic document.

6A-6I illustrate exemplary data structures useful for practicing at least some embodiments of the present invention.

7A-12B illustrate exemplary electronic documents and related exemplary data structures useful for executing electronic ink annotations of these electronic documents.

13A-14B illustrate exemplary electronic documents and related exemplary data structures useful for processing electronic flow chart features of the electronic documents.

15A-15C illustrate exemplary electronic documents and related exemplary data structures useful for processing electronic table features of the electronic documents.

16A-16D illustrate a method of processing an electronic ink in accordance with various embodiments of the present invention.

17, 19-21, and 23-26 illustrate the delivery of data objects during an exemplary ink analysis process in accordance with various embodiments of the present invention.

18 and 22 illustrate simple data trees that can be handled in accordance with various embodiments of the present invention.

FIG. 27 is a flowchart showing a method of adjusting analysis results of a current state of a document. FIG.

FIG. 28 illustrates an apparatus for asynchronous analysis of electronic ink in accordance with still other examples of the present invention. FIG.

Detailed description of the drawings

<Terms>

The following terms are used herein, otherwise specified or resolved from the description, the terms having the meanings provided below:

"Render" or "Rendered" or "Rendering"-The process of determining how information is to be delayed, printed on the screen, or output in some other way.

"Computer-Readable Medium"-any available medium that can be accessed by a user on a computer system. By way of example, and not limitation, “computer readable media” may include computer storage media and communication media. "Computer storage media" includes non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. "Computer storage medium" includes RAM, ROM, EEPROM, flash memory or other memory technology; CD-ROM, digital versatile disk (DVD) or other optical storage device; Magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices; Or any other medium that can be used to store desired information and can be accessed by a computer. A "communication medium" typically embodies other data or other transmission medium within a modulated data signal, such as computer readable instructions, data structures, program modules or carrier waves, and includes any information delivery medium. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as wired networks or direct-wired connections and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of "computer readable media".

<Overview>

According to various embodiments of the present invention, the attributes of an element in a document (or other file type) may include information about the spatial location of the element in the document. Therefore, both the electronic ink stroke and the typed text can be described by the same spatial coordinate system for the document and / or by spatial or other relationships with other elements in the document. In addition, related elements within a document can be identified simply by identifying the spatial area of the document containing the elements and / or by linking the elements.

By using this spatial relationship between various document elements, a software application can create and maintain a data structure that describes the different relationships between document elements. For example, a software application can maintain a data structure such as a data tree that defines classes for various elements in a document. Therefore, a node in the data structure may correspond to a handwritten word or picture containing one or more ink strokes, and the data stored at that node may also indicate the location of the word on the page. Any other suitable or conventional data is typically stored with respect to the ink stroke and / or electronic ink words may also be stored at the word node.

This type of data structure can also associate document elements (such as individual ink strokes or typed text characters) to meaningful groups such as words, lines of words, sentences, paragraphs, and the like. Thus, if a software application maintains a document tree structure describing a paragraph of handwritten electronic ink, the leaf nodes of that data structure may contain data relating to individual strokes of the electronic ink, The strokes may be associated together in the data structure as word nodes corresponding to words in the paragraph (eg, words determined by the parser and / or recognizer). The tree structure may then associate the word nodes with the line nodes corresponding to the lines in the paragraph. Each line node may also be associated with a node corresponding to that paragraph. The software application may also include non-ink document elements, such as images, for nodes corresponding to electronic ink strokes and / or several other groups of electronic ink; Ink painting; Typed characters, words, lines, paragraphs; It can maintain a tree or other data structure that associates with another node that corresponds to the same. Therefore, such data structures may be used to distinguish electronic ink strokes forming handwritten text from electronic ink strokes annotating non-ink document elements, and / or to link electronic ink strokes to other document elements. It can be used to define the relationships between associated electronic ink strokes.

As will be discussed in detail below, these data structures can be used as an ink analysis tool in accordance with various embodiments of the present invention for analyzing electronic ink in a document. According to various embodiments of the present invention, a software application may first obtain an analysis of electronic ink in a document by creating a data structure for that document. The data structure describes (if any) the relationship between document elements that have already been analyzed, and thus provides the context in which any new electronic ink will be analyzed. This data structure twisted "analysis context object" also includes any new electronic ink that is unanalyzed. That is, the analysis context object also includes an electronic ink that has not yet been established with other document elements. In some examples of the invention, the software application creates an analysis context object for itself. However, as other examples of the present invention, the software application uses an ink analysis tool or another tool to create an analysis context object.

After the software application provides the analysis context object to the ink analysis tool (or after the ink analysis tool creates the analysis context object), the ink analysis tool is informed about at least a portion of the analysis context object containing the unanalyzed electronic ink. Make a copy of, or retrieve that information. By making a copy of the information about the desired portion of the analysis context object or retrieving the information, the ink analysis tool can be analyzed next without changing the analysis context object maintained by the software application. That is, the copy is independent of the actual electronic document used in the software application and can therefore be referred to below as the "document independent" analysis context.

Once the ink analysis tool creates a document independent analysis context object, the ink analysis tool provides the document independent analysis context object to one or more analysis processes. For example, if handwriting recognition is performed on an unresolved electronic ink in a document, the ink analysis tool may classify the ink into text and picture strokes (if necessary or required) and then map the unresolved electronic ink text strokes to the ink layout. The document independent analysis context object may be provided with a classification step and / or layout analysis processes for grouping into related groups based on that. While the classification step and / or layout analysis processes analyze the document independent analysis context object, the software application may continue to operate regularly. In particular, the software application may continue to receive new electronic ink input and / or any other data in the electronic document continued by the application program.

When an analysis process such as a parsing process has completed the analysis of the document independent analysis context object, the analysis results are returned to the ink analysis tool. In particular, the parsing process (which may in particular include classification and layout analysis processes as described above) returns to a modified version of the document independent analysis context object that previously shows new relationships for unanalyzed electronic ink. Because the software application is free to accept any other data about the document during new electronic ink input and / or pre-written parsing operations, the current version of the analysis context target for the document (ie, the version maintained by the application) is The document independent analysis context object originally provided with the ink analysis tool and the parsing results provided by the parsing process may be different.

Thus, as some examples of the present invention, the ink analysis tool can obtain a current version of an analysis context object from a software application and adjust the parsing results with the current version of the analysis context object. During this adjustment process, the ink analysis tool will update the current version of the analysis context object to reflect the results of the parsing process. The ink analysis tool will then transfer the adjusted data from the current analysis context object to the handwriting recognition process for recognition. However, as other examples of the present invention, the ink analysis tool may omit the adjustment process, and instead pass the parsing results directly to the handwriting recognition process.

Once the parsing results have been adjusted to the current version of the analysis context object, the software application can return back to regular operation, thus continuing to receive new electronic ink input and / or any other data about the document. Can be. On the other hand, the recognition process analyzes the adjusted data (or alternatively its parsing results) from the current analysis context object. After the recognition process completes the analysis of the adjusted data (or parsing results), the recognition results are returned to the ink analysis tool. Again, because the software application may have received any other data and / or new electronic ink input for the document during the operation of the recognition process, the ink analysis tool obtains the current version of the analysis context object from the software application. do. The ink analysis tool then adjusts the results from the recognition process to the current version of the analysis context object to update the current version of the analysis context object with the adjustment results.

<Example Operating Environment>

1 shows a schematic diagram of a general-purpose digital computing environment that can be used to implement various embodiments of the present invention. In FIG. 1, computer 100 includes a system bus 130 that couples processing system 110, system memory 120, and various system elements, including system memory 120, to processing unit 110. Include. The system bus 130 can be any of several types of bus structures including a memory bus or a memory controller, a peripheral bus, and a local bus using any of the various bus architectures. System memory 120 may include read only memory (ROM) 140 and random access memory (RAM) 150.

Basic input / output system 160 (BIOS) includes basic routines that help to transfer information between elements in computer 100, such as stored in ROM 140 during startup. Computer 100 also includes a hard disk drive 170 for reading from and writing to a hard disk (not shown), and a magnet for reading from and writing to a removable magnetic disk 190. Disk drive 180 and optical disk drive 191 for reading from and writing to removable optical disk 192, such as a CD ROM or other optical media. The hard disk drive 170, the magnetic disk drive 180, and the optical disk drive 191 are connected to the system bus 130 by the hard disk drive interface 192, the magnetic disk drive interface 193, and the optical disk drive interface 194. Respectively). Such drives and their associated computer readable media provide non-volatile storage of computer readable instructions, data structures, program modules, and other data for the personal computer 100. Computer readable media capable of storing computer-accessible data such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read-only memories (ROMs), and the like. Those skilled in the art will understand that other types of can be used in the exemplary operating environment.

Many program modules that may be stored on hard disk drive 170, magnetic disk 190, ROM 140, or RAM 150 include operating system 195, one or more applications 196, other program modules ( 197 and program data 198. A user may enter commands and information into the computer 100 through input devices such as keyboard 101 and pointing device 102 (such as a mouse). Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 110 via a serial port interface 106 connected to the system bus 130, but they are also connected to other ports such as parallel ports, game ports, or universal serial buses (USB). Can be connected by interfaces. In addition, these devices may be coupled directly to the system bus 130 via a suitable interface (not shown).

The monitor 107 or other type of display device may also be connected to the system bus 130 via an interface such as the video adapter 108. In addition to the monitor 107, a personal computer typically includes other peripheral output devices (not shown) such as speakers and printers. In one example, a pen digitizer 165 and accompanying pen or stylus 166 are provided to digitally capture freehand input. Although the connection between the pen digitizer 165 and the serial port interface 106 is shown in FIG. 1, the pen digitizer 165 may in fact be directly coupled to the processing unit 110, or known in the art. As can be coupled to the processing unit 110 in any suitable manner, such as a parallel port or another interface and system bus 130. Moreover, although digitizer 165 is shown as separate from monitor 107 in FIG. 1, the usable input area of digitizer 165 may be the same area as the display area of monitor 107. Digitizer 165 may also be integrated into monitor 107, or may exist as an overlapping discrete device, or otherwise added to monitor 107.

Computer 100 may operate in a network environment using logical connections to one or more remote computers, such as remote computer 109. The remote computer 109 can be a server, router, network PC, peer device or other conventional network node, although for simplicity only the memory storage device 111 is shown in FIG. This typically includes many or all of the elements described above for the computer 100. The logical connections shown in FIG. 1 include a local area network (LAN) 112 and a wide area network (WAN) 113. Such networking environments are commonplace in offices, large optical computer networks, intranets and the Internet using both wired and wireless connections.

When used in a LAN networking environment, the computer 100 is connected to the local area network 112 via a network interface or adapter 114. When used in a WAN networking environment, the personal computer 100 typically includes a modem 115 or other means for establishing a communication link over a wide area network 113, such as the Internet. The modem 115, which may be inside or outside the computer 100, may be connected to the system bus 130 via the serial port interface 106. In a networked environment, program modules depicted for the personal computer 100 or portions thereof may be stored in the remote memory storage device.

It is to be understood that the network connections shown are examples and that other techniques for establishing a communication link between the computers can be used. It is assumed that any of a number of well known protocols exist, such as TCP / IP, Ethernet, FTP, HTTP, UDP, etc., and the system is user-server to allow the user to retrieve web pages from a web-based server. Can be operated in a configuration. Any of a variety of conventional web browsers can be used to display and manipulate data on web pages.

Although the environment of FIG. 1 illustrates an example environment, it should be understood that other computing environments may also be used. For example, one or more examples of the invention use fewer environments than all of the various embodiments shown in FIG. 1 and described above, which embodiments will appear in various combinations and subcombinations apparent to those skilled in the art.

2 illustrates a pen-input personal computer (PC) 201 that may be used in accordance with various embodiments of the present invention. Any or all of the features, subsystems, and functions of the system of FIG. 1 may be included in the computer of FIG. 2. The pen input personal computer system 201 includes a large display surface 202 on which a plurality of windows 203 are displayed, for example a digitizing flat panel display such as a liquid crystal display (LCD) screen. Using the stylus 204, the user can select, highlight, and record on the digitizing display area. Examples of suitable digitizing display panels include electromagnetic pen digitizers such as Mutoh Co. (now known as FinePoint Innovations Co.) or pen digitizers available from Wacom Technology Co. Other types of pen digitizers may also be used, such as optical digitizers and touch-sensitive digitizers. The pen-input computing system 201 uses gestures using the stylus 204 to perform conventional application tasks such as manipulating data, entering text, creating, editing, and modifying spreadsheets, word processing programs, and the like. Interpret (gesture)

The stylus 204 may be equipped with buttons or other features to improve its performance. In one example, the stylus 204 may be embodied as a "pencil" or "pen", with one end constituting the recording portion and the other end moving for the display, where the electronic ink portions on the display to be erased. Make up the "Eraser" part to display. Other portions of input devices such as a mouse, trackball, keyboard, etc. can also be used. In addition, the user's own finger may be used to select or display portions of the image displayed on a contact or proximity-sensitive display. As a result, the term "user input device" as used herein is intended to have a broad definition and encompasses many variations of well known input devices.

In various examples, the system provides an ink platform as a set of COM (Component Target Model) services that an application can use to capture, manipulate, and store ink. The ink platform may also include a markup language that includes a language, such as Extensible Markup Language (XML). The system can also use DCOM as another implementation. In addition, implementations including the Win32 programming model and the Net programming model from Microsoft Corporation may be used. Such platforms are commercially available and known in the art.

In addition to the use of full-performance pen-type computing systems or "tablet PCs" (eg, convertible laptops or "slate" type tablet PCs), embodiments of the present invention may be hand-held or palm. Palm-top computing systems; A personal digital assistant (PDA); A pocket personal computer; Mobile and cellular telephones, pagers, and other communication devices; With a clock; Home appliances; Any other devices or systems, including a monitor or other display device, and / or present printed or graphical information to users and / or allow input using an electronic pen or stylus or by another device Other types of pen-input computing systems, such as digitizers that can process the collected electronic ink (eg, conventional desktop computers that can process the collected electronic ink by a tablet PC) and / or electronic It may be used in other devices that accept data as ink and / or accept an electronic pen or stylus input.

The invention will now be described with reference to the various embodiments of the invention and the remaining figures showing information for aiding the description. The specific drawings and information contained in the detailed description should not be inferred as limiting the invention.

Spatial Document Viewing Abstraction

As described above, some embodiments of the present invention generally relate to systems and methods for providing richer and more versatile annotations using electronic ink in electronic documents. The following describes various embodiments and examples of the present invention in more detail.

A. Comprehensive Overview of the Invention

3 generally illustrates the operation of a system and method in accordance with at least some examples of the present invention. In particular, as shown in FIG. 3, a user can interact with electronic document 300 (document “A” in FIG. 3) in several ways, for example, by adding electronic ink to electronic document 300. . In the illustrated example, the electronic document 300 includes electronic or typed text 302 (" Sample Text " in the illustrated example) and ink drawing 304 (home in the illustrated example). Of course, the electronic document 300, such as electronic text, electronic ink (picture or text), images, graphs, tables, charts, other graphics or electronic information and / or combinations thereof, does not deviate from any data or information in the present invention. It may include. For the purposes of this example, its first electronic document 300 (also referred to herein as a “reference document” or “reference portion”) includes electronic typed text 302 and electronic ink picture 304. As the user reviews document A 300 in this example, she decides to include electronic ink annotation within document 300. In particular, in the illustrated example, the annotation includes a circle 306 of margin 308 of the portion of the electronic text 302.

In this example, after the user enters the annotation, the application will call the parser 310 of the newly entered electronic ink data and request parsing (as mentioned earlier, "parsing" is for example another In addition to analytical processes, the classification and / or classification of electronic ink into various ink types (e.g., pictures, text, flowcharts, music, etc.) and / or (e.g., spatial and positional relationships between ink strokes and Ink layout analysis, grouping into appropriate groups). In particular, the application program will send the input electronic ink data to the parser 310 as unclassified ink data. This is illustrated by input arrow 312 in FIG.

Optionally, the application program may simultaneously send some or all data about the underlying electronic document 300 to the parser 310. Such data may include, for example, data regarding the spatial layout of information within document 300, such as locations of electronic text 302, picture 304, and the like. In addition, the application can send data representing the hierarchical structure of the data in the document, as described in more detail below. As another alternative, and also described in more detail below, parser 310 may store any data about electronic document 300 (in this case, such as information about electronic document 300 in a spatial area associated with a new unclassified ink). Call back to an application that requests spatially typed text and data representing information). This data input to parser 310 is shown by input arrow 314 in FIG. 3. Of course, both unclassified ink and pre-analyzed data can be sent to the parser simultaneously and / or made available to the parser without departing from the present invention.

The parser 310 takes the inputted electronic document data 314 and inputted unclassified ink data 312 and uses that information to classify new electronic ink data into ink types. The ink can be classified into various other ink types without departing from the present invention, and various examples of possible ink types are described in more detail below. For this example based on the input ink 306 and the information 302 included in the underlying document 300, the parser 310 according to this example of the present invention may annotate or more particularly specify the input ink 306. It can be determined and classified as an ink picture corresponding to a container annotation (e.g., a circle) containing typed text. Of course, keynote annotations, "highlight as" comments, "highlight from" comments, (e.g. anchor comments without or highlights from, horizontal span, vertical range) Other annotation types such as (vertical span) are also possible, and various examples can be described in more detail below.

Once parsed, the data associated with the annotation 306 can be incorporated into a hierarchical data structure (or any other suitable or desired data structure) of the "document tree" associated with the electronic document 300, and the data structure ( Or instructions for changing an existing data structure maintained by the application program may be returned to the application program as shown by arrow 316 in FIG. 3. If desired, other data may also be returned to the application program at parser 310 without departing from the present invention. For example, as shown in FIG. 3, if the operation of the included parser 310 calls one or more recognizer modules, components, and / or programs, recognition (such as a handwriting recognizer or other recognizer). Data associated with the results of the analysis process can be returned to the application.

4 shows an example of an implementation 400 of various features used in some examples of this invention. As shown, an electronic document (such as document A from FIG. 3) is created, stored, and / or maintained by the application 400a of the neuter structure or document model 402. This data structure 402 may be a hierarchical data structure or any other desired or suitable data structure without departing from the present invention. Application 400a includes an "analysis context" object 404, which includes a mirror copy, an optional copy, or other relevant information about the data structure 402 (eg, analysis context object 404). ) May be a "translation" layer that queries the analysis context structure 404 translated into queries of a native document mode structure-in this way, according to embodiments of the present invention. Parsing techniques can be translated for use with many other native applications). The operating system side of the "platform" 400b or implementation of this example 400 includes an "ink analyzer" method 406, which is called by the application 400a. The application 400a includes an ink analyzer method 406, which references and passes the analysis context object 404 therein. The ink analyzer method 406 will call the parser 408 (also part of the platform 400b in this example), which classifies and analyzes the layout of the input data as described above comprehensively. Example implementation operations in accordance with the present invention are described in more detail in the <Analyze Ink> section below. Once the parser 408 has completed the analysis and processing, the platform 400b is document document 402 and / or analysis context object 404, as illustrated comprehensively by arrow 410 in FIG. 4. The data can be returned to the application 400a to reconstruct its data structure based on this parser processing and results.

FIG. 5 illustrates an electronic device in at least some examples of the present invention, for example, a data structure generated and / or output by a portion of an “analysis context” object 404, a document model 402, or a parser 408. A comprehensive illustration of an example of a data structure that can be used to store document data is shown. In particular, in FIG. 5, an example electronic document 500 is shown. When the data structure is stored as an analysis context object 502 or targeted for parsing, the data structure may include various document elements that may be classified or arranged into different groupings or "nodes". . For example, as shown in FIG. 5, the electronic document 500 includes three electronic text paragraphs 504, 506, and 508; One electronic ink drawing 510; One electronic ink tin 512 is included. An example analysis context data structure 502 for an electronic document 500 is shown in FIG. 5 as a list of high-level elements. In particular, node element 504 (a) corresponds to paragraph 504 in electronic document 500, node element 506 (a) corresponds to paragraph 506 in electronic document 500, and node element 508 (a) corresponds to paragraph 508 in the electronic document 500, and node element 510 (a) corresponds to paragraph 510 in the electronic document 500, and node element 512 (a ) Corresponds to paragraph 512 in electronic document 500. Additional nodes, as will be described in more detail below, the higher level nodes 504 (a), 506 (a), 508 (a), 510 (a), 512 (a) shown in FIG. Can be stored in a data structure (eg, in a hierarchical array) as a parent node and / or child node.

B. Context Nodes and Hierarchical Data Structures

6A-6I illustrate various examples of context nodes and data structures that can be used to construct, analyze, and parse electronic document data, such as an analysis context object, in accordance with some examples of the present invention. 6A illustrates an example data structure 600 that may be used to store unclassified electronic ink data (eg, ink newly entered by a user who has not previously been targeted for parsing). The data structure 600 of this example is a root node (which may correspond to an entire transcription document in any suitable or desired group of data, such as an application, a particular page of a document, or a group used by the application). node) 602. Any suitable or desired data may be included in the root node 602, such as pointers to the number of pages, previous and next pages, location or size of margin, and the like. Each root node 602 can include any number of unclassified ink nodes 604 as child nodes, and each unclassified ink node 604 can also include any number of individual stroke nodes 606. (The last node in the hierarchy (in this example, the stroke node 606 may also be referred to herein as a "leaf node"). (The letter "n" in the various figures. Represents any number, including 0.) The various nodes of data structure 600 may contain any desired or appropriate data corresponding to that node. A digitizer point or other data identifying an ink stroke; stroke position or orientation data; stroke color data; stroke pressure data; stroke input timing data; stroke ID data; and / or a system capable of receiving electronic ink; May include data indicative of any other conventional or useful data used to store the stroke information in the law Similarly, the unclassified ink node (s) 604 may: bounding boxes of all strokes contained within the node box) size; bounding box position; input timing information; geometric information about the ink (e.g., minimum convex hull information); and / or unclassified ink strokes contained in the node. Any desired or appropriate data associated with the ink or node, such as data representing any other conventional or useful data relating to the alternative, alternatively, as needed, the stroke data is generally more than a separate node in the document tree. It may be stored as another node (eg, word node, table node, etc.) attribute.

Once the unclassified ink has been sent to the classifier, layout analyzer, recognizer, and / or another parsing system, at least some of the ink may be associated together and / or classified into various other types and associated with that ink. The data structure can be changed to correspond to the various combinations and types in which it has been classified. 6B shows an example data structure 610 for an electronic ink that has been classified as ink text. Using parsing techniques, including conventional parsers known and used in the art, input electronic ink strokes (e.g., pen-down events and the following pen-up events in time) -up event) or electronic ink data collected in some other manner) can be parsed and stored in a hierarchical manner, and the individual ink strokes involved can be grouped together and stored as ink words, and the linearly placed ink The words may be grouped together and stored as ink lines, the associated ink lines may be grouped together and stored as ink paragraphs and the related paragraphs may be electronic documents (eg, as individual page parts, as prerequisite documents, or To any suitable root group). As shown in FIG. 6B, an example hierarchical data structure 610 corresponding to electronic ink data parsed and stored in this manner may include a root node 612, which may be an entire electronic document, page, or application. Corresponds to another group used by the program. Each root node 612 may include any number of nodes 614 corresponding to shorts or similar groups of electronic ink strokes. Shorting nodes 614 may include any number of individual line nodes 616, which also includes any number of individual stroke nodes 620 corresponding to individual strokes of input electronic ink data. can do. Optionally, as described above, the stroke data may be stored as "attributes" of the associated word node, as needed. The various nodes 612, 614, 616, 618, 620 may contain a collection of various individual strokes or strokes contained within the node, such as similar spatial orientation, or any suitable data regarding positional data and / or other data as described above. Can be stored.

Other data types and data structure arrangements corresponding to the input electronic ink strokes are possible without departing from the present invention. For example, classification, layout analysis, or recognizer techniques may determine that the input electronic ink forms an ink picture. 3 shows an example ink drawing 304 that includes several individual ink strokes. 6C and 6D show exemplary hierarchical data structures that can be used to group and store electronic ink data determined to relate to ink pictures. As shown in FIG. 6C, the data structure 630 includes a root node 632 (which may correspond to an entire electronic document, an electronic page, or some other group of data, as described above). Each root node 632 can be any number of individual (eg, each individual picture has individual nodes that store data, for example, picture width, picture height, picture position, bounding box size, etc.). Ink picture nodes 636, each ink picture node 636 includes any number of individual ink stroke nodes 638 (corresponding to the individual ink strokes that make up the picture). can do. Optionally, the ink stroke data may be stored as the "attributes" of the corresponding ink picture node (or other node).

6D shows an alternative data structure 630a for electronic documents that include one or more ink pictures. In this example, an electronic document or page or other group of data (denoted as root node 632a) is associated with one or more groups (node 634a) having their data stored in a hierarchical manner described in connection with FIG. 6C. (E.g., an entire electronic document (denoted as root node 632a) may comprise several individual pages (each denoted as group node 634a)), each individual page being one or more individual Ink pictures (represented individually as ink picture node 636a), and each individual ink picture may include one or more individual ink strokes (represented as stroke node 638a or individually as attributes of the ink picture node). . It should be understood that the two groups and the ink pictures can be children of the same root node or any group node. Also, one group node may include additional group nodes as needed. Of course, other suitable data structures for storing electronic ink pictures can be used without departing from the present invention.

However, hierarchical data structures are not limited to use as input electronic ink data. Input typed text (eg, from a keyboard) can also be associated with various groups and stored in a hierarchical data structure. 6E shows an example of such a data structure 640. This data structure 640 may include a root node 642 (eg, such as the root nodes discussed above). Each root node 642 may include any number of text paragraph nodes 644 (corresponding to paragraphs of text), and each paragraph node 644 may include any number of text line nodes. 646 (or alternatively text sentence nodes), and each line node 646 may further include any number of individual text word nodes 648. If desired, text words can also be further broken down into individual text character nodes without departing from the invention (eg, baseline, thin line, ascender and descender shapes of each character). Can be further broken into individual glyph nodes). Various nodes include text such as paragraph, line or word spatial location, content, size, etc .; Page margins; Margin size or location; Any suitable data about the number of pages and the like can be stored.

The various data structures described above with respect to FIGS. 6A-6E illustrate data structures that exclusively include electronic ink data or only electronic typed text data. Parsing techniques can be used to analyze, combine, associate, and group these different types within a single data structure without departing from the invention. 6F illustrates that an electronic document or portion thereof (denoted as root node 652) includes one or more individual paragraphs (denoted by paragraph node 654) and one or more individual lines (denoted by line node 656). An example data structure is shown. The lines can include both potentially typed text (denoted by text word node 658) and electronic ink text (denoted by ink word node 660 and ink stroke node 662). Each individual paragraph, line and word node (both typed text and ink words) will contain appropriate independent data, depending on the content of the paragraph, line and word. Additionally or alternatively, the individual words may further include both ink letters and typed letters without departing from the present invention.

Other electronic data can also be included in the data structure of an electronic document (including hierarchical data structures) without departing from the invention. For example, image data (such as digital photos, graphic information, etc.) may be included in the data structure for the electronic document, as shown, for example, in data structure 664 of FIG. 6G. As shown in this example, the data structure 664 includes an electronic document or portion thereof (denoted as root node 666) that includes one or more individual groups of data (denoted as group node 668). do. Each group of data in this example is typed text (represented by paragraph node 672, line node 674, and text word node 676), as previously described above with respect to FIGS. 6E and 6F. As well as an electronic image (represented by image node 670). Of course, a data structure such as shown in FIG. 6G may also include electronic ink data (eg, such as the data structure shown in FIGS. 6A-6D) or other desired data type without departing from the present invention. 6H illustrates an example of another potential data structure 680 that may be used in accordance with at least some examples of the present invention. In particular, its data structure 680 in FIG. 6H corresponds to an electronic document or portion thereof (indicated as root node 682) that includes one or more lists (indicated by list node 684). The list in the electronic document may include any number of individual list items (represented by list item node 686), and each list item may contain a wide variety of different types of electronic information. For example, some list items may optionally include a list bullet, which may consist of a typed bullet or an electronic ink bullet (containing one or more corresponding ink strokes 690). Ink bullet node 688 is shown in the example of FIG. 6H). In addition, each list item may optionally include one or more electronic ink paragraphs (denoted by node chain 692), one or more ink pictures (denoted by node chain 694), and / or one or more typed paragraphs. (Denoted by node chain 696). Additionally (or alternatively), each list item may optionally include one or more images (represented by image node 698) and / or one or more data groups, such as the groups described above with respect to FIGS. 6D and 6G. (Denoted by the group node chain 700). Various nodes and / or node chains 692, 694, 696, 698, 700 generally correspond to the portions and / or groups of such data structures as well as the example data structures shown in FIGS. 6A-6G. As such, other descriptions are omitted. If desired, in at least some cases, the individual list item node 686 itself may also include a list having a data structure as shown in FIG. 6H.

6I illustrates another exemplary data structure 710 that can be used in at least some examples of the present invention. This example data structure 710 is analyzed as a table, associated together, grouped, and corresponds to stored electronic data. In this illustrated example, each electronic document or portion thereof (denoted as root node 712) may include one or more tables (denoted as table node 714). In this example, the table may consist of one or more rows of information (represented by row node 716), and every row may consist of one or more individual cells (represented by cell node 718). The individual cells may comprise one or more data structures, including for example any of the various other data structures described above. In the example shown, the individual table cells are any number of images (represented by image node 720), any number of typed text paragraphs (represented by node chain 722), any number of Potentially including groups (represented by node chain 724), any number of additional tables (represented by node chain 728), and any number of lists (represented by node chain 730). Is shown. Of course, various specific data structures described above and / or any other suitable or desired data structures, including, for example, combinations and portions of data structures such as those described above, may be included in the table without departing from the invention. .

Any suitable or desired data may be stored at various other nodes, including the various types of data described above. For use in processing data relating to annotations, it may be useful in at least some examples of the invention for various nodes to store data about the spatial position, orientation, or position of a node in the electronic document, optional This spatial information in turn relates to or links with other nodes in the electronic document data structure.

In addition, other types of parsers, classifiers and / or recognizers and data types may be used without departing from the present invention. For example, recognizers can be used without departing from the present invention, but with electrical or electronic symbols (eg, resistors, voltage sources, capacitors, etc.); Music symbols; Mathematical symbols; Flowchart elements; Pie chart elements and the like can be used to analyze, recognize, group, and / or associate.

The data structures shown in FIGS. 6A-6I are simple examples of various data structures that can be used in accordance with embodiments of the present invention. Those skilled in the art will recognize that the specific data structures used may be widely modified without departing from the invention. For example, other nodes representing different layout groups may be used, additional node types may be added, or some of the described node types may be omitted without departing from the present invention (eg, column nodes are tables May be used instead of row nodes within the data structure). In addition, the description is optimized for use in languages and textual representations in which characters are read from left to right and top to bottom, page by page. Data structures that correspond to and use other languages and character placement, for example, data structures to accommodate languages read and / or written in right to left, top to bottom, top to bottom and combinations thereof. Can be used without departing from the present invention. In addition, if desired (especially for some languages, such as Asian languages), the strokes can be properly grouped into letters, and then into lines without using an intermediate "word" node, as described above. Can be grouped. Alternatively, the use of line nodes can be omitted and word nodes can be grouped together as paragraphs and / or other desired groups. These and other changes in data structure may be used without departing from the invention to accommodate characters of a particular language.

C. Application of Embodiments of the Invention to Comments

Embodiments of the present invention, including the use of context nodes and hierarchical data structures as described above, can be used to provide "smart" electronic ink annotations in electronic documents. As described above, pen-type computing systems are preferably and advantageously used for annotation of electronic documents. However, in order to be particularly useful and effective, annotations made of electronic ink in documents on a pen-type computing system may be used for reasons related to various elements in the underlying document (eg, margin change, display size change, font). Change, add information, delete information, etc.), the annotation should be flexible enough to accurately change position and / or other features based on changes to annotated elements in the underlying document. This may be accomplished, for example, by linking at least one context node associated with the annotation to one or more context nodes associated with the underlined document. In at least some examples, the data and / or properties associated with the document of the base and stored with respect to the document of the base, such as spatial properties locations, and / or data about the location of one or more document elements, may be the location, shape of the annotation. And / or to control other properties.

7A and 7B show an example of annotation that can be applied to an electronic document through electronic ink. As shown in FIG. 7A, the electronic document 750 includes the sentence “Today the sky is green”. A user of the pen-input computing system annotated this document 750 using electronic ink to underline the word "green" (the underlined annotation is shown by reference numeral 752). 7B shows an example hierarchical data structure 760 corresponding to such a compound electronic document 750 (the entire electronic document 750 is represented within the data structure by the root node 762. Once fully Once parsed, the data structure 760 will include context nodes that show the entire document structure, in particular, information about typed text within the electronic document 750 indicates that the electronic document 750 is a paragraph. (Paragraph node 764), the paragraph containing a line (line node 766), and that line contains five words (each of the words "Today", "the", "sky", " will be analyzed and associated together to indicate that it contains one text word node 768a through 768e for " is " and " green. " Note the presence of data in 750), and the data Pictures will be classified as ink picture node 770. Additionally, based on the location and location of the ink picture stroke (s) for typed text within the underlying document 750, the parser will render its annotations. It will recognize and classify it as an "underlined" annotation, and will generate an underline node 772. Additionally, the parser will actually contain the underlined annotation in one or more strokes (leaf node 774). All stroke data corresponding to the ink stroke (s) will be stored (optionally, the stroke data can be stored as an attribute of the underline node 772 or another suitable node).

Additionally, whenever an individual word "green" is in the electronic document 750 (eg, whenever the word has to be moved for some reason), the electronic ink underline stroke 752 is replaced by the individual word "green". In order to maintain, the underline context node 772 (eg, the "source" context node in this link example) is the text word node for the word "green" (as shown by arrow 776 in FIG. 7B). Data is stored in the data structure 760 of this example to indicate that it is linked to 768e, and the data is stored to indicate that the text word node 768e is linked to the underline node 772 as indicated by the arrow 778. do. This link anchors its underline node 772 to the text word node 768e (eg, an “anchor” type link). Thus, if for some reason the data stored in the text word context node 768e indicates that the position of the word "green" has been changed in the electronic document, the links 776, 778 indicate the underlined annotation, if possible by the application. Will allow the detection of the need to move the rendering of to the point and / or spatial location of the text word context node 768e in the modified electronic document. As another example, "bounding box" data, word width data, or other data stored in context node 768e may be for some reason (e.g. changing font size, adding characters, deleting characters, changing characters, etc.). Indicative of a word size or position change, the application may possibly modify the size of the rendered underline to correspond to the new size associated with the text word represented by the context node 768e, if possible. Stretch or cut). If a system and method in accordance with at least some examples of the invention applicable is capable of breaking the stroke (s) of the comment to represent more than one line (eg, due to added words, hyphenation, etc.), Changes to the underlined document cause the annotated document element to be on multiple lines. In addition, any suitable link placement or link between nodes may be used without departing from the present invention.

8A and 8B show another annotation example. In this example, as shown in FIG. 8A, the electronic document 800 again includes the sentence “Today the sky is green”, but in this case, a user of the pen-input computing system strikes the word “green”. The document 800 has been annotated using electronic ink to mark out (the strike-out is shown by reference 802). FIG. 8B shows an example hierarchical data structure 810 corresponding to this compound electronic document 800 when fully parsed (because many of the specific context nodes of FIG. 8B correspond to context nodes present in FIG. 7B). , The same reference numerals as those present in FIG. 7B are used in FIG. 8B, and redundant description is omitted). In this case, the parser will note the presence of data in the electronic document 800 for annotation 802 and will classify this data as an electronic ink picture (ink picture node 812). Additionally, based on the location and location of the ink picture stroke (s) relative to the content of the underlying typed text in document 800 (eg, the strikeout spans the text word horizontally). The parser will recognize and classify the annotation as a "strike-out" annotation and create a strikeout node 814. Additionally, the parser sends one or more stroke leaf nodes 816 with all the stroke data corresponding to the actual ink stroke (s) of the strikeout annotation (or, alternatively, strikeout node 814 or another appropriate). As a node's attributes).

As with the underline annotation described with respect to FIGS. 7A and 7B, if the word has to be moved for some reason, the strikeout annotation may be retained in the individual word "green". Again, this means that the strikeout context node 814 (in this example, the "source" context node) is the text word node 768e for the word "green", as shown, for example, by the arrow 818 in FIG. 8B. Data in the data structure 810 to indicate that it is linked to the " destination " context node in this example, and the text word node 768e, as indicated by the arrow 820, is a strikeout node 814. By storing data in a data structure to indicate that it is linked to. The link type of this example is a " horizontally spanning " link type that anchors its strikeout node 814 to the text word node 768e. Thus, if the data stored in text word context node 768e indicating the location and / or size of the word "green" (or other word (s) associated with node 768e) has changed in document 800 for some reason, The links 818, 820 may adjust the rendering of the strikeout, if possible, and / or (if possible) the size, spatial position, and / or position of the word stored in the context node 768e in the changed electronic document. Or it will allow you to detect if you need to resize. Again, any suitable link placement or link between the nodes may be used without departing from the present invention.

9A and 9B illustrate another example of the type of annotation, further illustrating the association or linking of the annotation with one or more text words. In the example shown in FIGS. 9A and 9B, the electronic document 900 again includes the sentence “Today the sky is green”, but in this case, a user of the pen-input computing system uses the words “is green”. Annotated document 900 using electronic ink for circle display (annotation of that circle or “container” type is shown at 902 in FIG. 9A). FIG. 9B shows an example hierarchical data structure 910 corresponding to this composite electronic document 900 when fully parsed (because many of the specific context nodes of FIG. 9B correspond to the context nodes present in FIG. 7B). , The same reference numerals as those present in FIG. 7B are used in FIG. 9B, and redundant descriptions are omitted). In this case, the parser will again note the presence of data in the electronic document 900 for annotation 902, and classify this data as an electronic ink picture (ink picture node 912). Additionally, based on the location and location of the ink glyph stroke (s) for the content of the underlying typed text in document 900, the parser recognizes and classifies the annotation as a "container" annotation. And create container node 914 in data structure 910. In addition, the parser sends one or more stroke leaf nodes 916 with all the stroke data corresponding to the actual ink stroke (s) of the container annotation (or, alternatively, as an attribute of an ink container node or another suitable node). Will save).

If for some reason the container annotation is to be moved, such as for the underline and strikeout annotations described with respect to FIGS. 7A-8B, the container annotation may be retained in the word “is green”. Again, this means that the ink container context node 914 ("source" context node in this example) is a text word node for the words "is green," for example, as shown by arrow 918a in FIG. 9B. It may be accomplished by storing data in its data structure 910 to indicate that it is linked to each of the fields 768d, 768e (in this example, a “destination” context node). Additionally, as indicated by arrows 920a and 920b in FIG. 9B, data may be stored in data structure 910 to indicate that text word nodes 768d and 768e are linked to container node 914. have. This link is a "containment" type of link. Thus, for some reason the data stored in the text word context nodes 768d and 768e representing the position and / or size of the words “is green” (or other word (s) associated with these context nodes) is for some reason electronic document 900. ), The links 918a, 918b, 920a, and 920b adjust the rendering of container annotations, if possible, and / or position the words stored in the context nodes 768d and 768e in the changed electronic document. It will allow to detect if the size, spatial location, and / or size should be adjusted. Additionally, if the user adds words between the words "is" and "green", or if the words or letters are removed or altered in any way, the application can add the container to accommodate these changes. You can modify the size and / or position of the annotation. Again, other link arrangements or data representing the link can be used without departing from the present invention.

10A and 10B show another type of comment that is commonly used, in particular a comment of the "margin comment" type. In this example, as shown in FIG. 10A, the user annotated the electronic document 1000 containing the sentence “Today the sky is green” with an electronic ink margin comment 1002 stating “it's not green!”. . Obviously, in this case, as shown in FIG. 10B, since this text exists in the electronic document 1000 as two lines (unlike one line shown in FIGS. 7A, 8A, 9A), The data structure 1010 associated with the typed text is changed. Thus, the data structure 1010 has two line nodes 1012 and 1014, the first line node 1012 being text word nodes 768a (associated with the words "Today" and "the"). 768b, and the second line node 1014 includes text word nodes 768c, 768d, 768e (associated with the words “sky”, “is”, and “green”).

Once the composite electronic document 1000 is fully parsed, the parser will recognize the annotation 1002 as containing electronic ink text, which may be interpreted (eg, due to the position of the margin of the document 1000). ) Will be classified as a comment of type "margin comment" Thus, the parser will create an appropriate context node 1016 for margin comments. Since this margin comment 1002 only contains the electronic ink text, the parser includes a paragraph node 1018, two line nodes 1020, 1022, and appropriate ink word nodes associated with the hierarchical structure of the electronic ink. (1024, 1026, 1028) and stroke nodes 1030, 1032, 1034) (as mentioned above, optionally and alternatively, the ink stroke data is an attribute of the ink word node (s). And / or its line nodes may be omitted from data structure 1010).

Another link relationship is shown in the example of FIGS. 10A and 10B. In particular, in this example, based on the page margin and position and orientation of margin comment 1002 relative to the underlying document content, the parser is coupled with margin comment 1002, as shown by arrow 1036 in FIG. 10B. An associated paragraph node 1018 (in this example a "source" context node) is linked to a paragraph node 764 (in this example a "destination" context node) associated with the typed text. In addition, the parser may indicate that paragraph node 764 associated with typed text is linked by paragraph node 1018 of margin comment 1002, as shown by arrow 1038. 1010) will store the data. This is a "anchored" type of link placement in which an ink paragraph is anchored to a text paragraph. In this way, every time a typed paragraph moves within the electronic document 1000, the links 1036, 1038 may optionally add additional lines and / or words to the paragraph, if the application is capable of it. Move the margin comment 1002 so that the lines and / or words are removed from the paragraph and other changes are made in the paragraph or electronic document 1000, but remain in the margin adjacent to the linked typed paragraph. It will allow you to detect that it should be turned on. Of course, other link arrangements or data associated with the links can be stored without departing from the present invention. For example, the margin comment node 1016 may be linked to one or more individual text word nodes of a typed paragraph, such as the first text word node 768a.

11A and 11B illustrate somewhat more complex examples of annotations recognizable by methods and systems in accordance with at least some examples of the present invention. In the example shown, the electronic document 1100 again includes the phrase "Today the sky is green", but in this case, the user annotated the electronic document 1100 using a combination of other annotation types. In particular, the user has a container type annotation 1102 surrounding the word "green", a margin comment type annotation 1104 comprising the electronic ink word "Blue!" And a margin comment annotation 1104 from the container type annotation 1102. A connector type tin 1106 is indicated.

Because of the relevant points and spatial orientation of the various annotation types for the underlying document, typed document text, and margins of the document, a parser system according to at least some examples of the present invention provides various annotation types, their content, and relative to each other. It will recognize their relationship and the electronic document typed text as described above comprehensively. An exemplary hierarchical data structure 1110 for the electronic document 1100 generated by the parser is shown in FIG. 11B. Since the hierarchical structure for typed text is the same as described previously in FIG. 7B, the same reference numerals are used in the structure of FIG. 11B, and the detailed description is omitted. Also, as shown in FIG. 11B, the node chain 1112 for ink container type tin 1102 is similar to that shown in FIG. 9B, and the node chain 1114 for most magnetic comment type tin 1104 is shown in FIG. Similar to that shown in FIG. 10B. Thus, detailed descriptions of these node chains are not described.

In order to maintain the spatial location of the annotations at the appropriate point and spatial location of the underlying text, various links between the annotation nodes and various links with appropriate text nodes are provided by the parser and stored in the data structure 1110. . In the example shown, as shown by arrow 1118 in FIG. 11B, the ink container node 1116 (here “source” context node) is linked to the text word node 768e of the word it contains. Similarly, as indicated by arrow 1120 in FIG. 11B, data is stored in data structure 1110 indicating that text word node 768e is linked to ink container node 1116. This is a "container" type link placement.

As mentioned above, the parser according to this example of the present invention also has an ink picture 1106 whose electronic document 1100 corresponds to an ink connector (i.e., container annotation 1102 and margin comment annotation 1104 in this example). (The arrows between)). This connector annotation 1106 may assign data representing the stroke or strokes that make up the connector (eg, to attributes associated with the stroke node (s) 1126 or ink connector node 1124 (or another suitable node). ) Is further stored in the data structure 1110 as an ink picture (indicated by ink picture node 1122 in this example), including a node indicating an annotation type (ink connector node 1124 in this example). Moreover, information is stored in the data structure 1110 to indicate that the ink connector node 1124 (the "destination" node in this example) is directed from that ink connector node 1116 (the "source" node in this example). This link (link of type “points from”) is indicated by arrow 1128 in FIG. 11B. In addition, the data structure 1110 displays data to indicate that the ink connector node 1124 (the "source" context node in this example) is linked to the ink word node 1132 (the "destination" context node in this example). It includes more. This link (link of type “point to”) is indicated by arrow 1134 in FIG. 11B. Alternatively, the ink connector node 1124 can be linked to any of the ink line node, ink shorting node, or margin comment node of the node chain 1114 without departing from the present invention. With the specific links described above, its data structure 1110 indicates that the ink connector 1106 points out of the circle picture 1102 and in ink words, as shown in FIG. 11A. These various links allow the application to detect the relationship between the underlying document and various annotation elements. Then, if possible, the application program can move the annotation elements to appropriately correspond to the movements in the underlying document. Any suitable link placement or data associated with a link can be used without departing from the present invention.

12A and 12B show another example of a commonly used tin shape. In this example, as shown in FIG. 12A, the electronic document 1200 again includes the sentence “Today the sky is green”, but in this case, the user of the pen-input computing system is near the first word of the sentence. Annotated the document 1200 using electronic ink to place a star or an asterisk in (the annotation is shown by reference number 1202). FIG. 12B shows an example hierarchical data structure 1210 corresponding to such a compound electronic document 1200 when fully parsed (since many context nodes of FIG. 12B correspond to context nodes present in FIG. 7B, FIG. The same reference numerals as those present in 7b are used in FIG. 12b and duplicate descriptions are omitted). In this case, the parser will note the presence of electronic ink data relating to the annotation and will classify this data as ink picture (ink picture node 1212). Additionally, based on the ink stroke shape as well as the location and point of the ink stroke (s) of the annotation 1202 relative to the content of the underlying document 1200 (e.g., alongside lines of typed text). , The parser will recognize the annotation as a "star", "bullet" or anchor type annotation, and create an asterisk node 1214. Additionally, the parser will store all the stroke data corresponding to the star's actual ink stroke (s) in one or more stroke leaf nodes 1216 or attributes (or some other node) of the star node.

In this example, if the typed text has to be moved for some reason, the asterisk type annotation 1202 may be maintained in the entire line of typed text whenever the line is in the electronic document 1200. This is, for example, a data structure to indicate that node 1214 (the "source" context node in this example) is linked to text line node 766, as shown by arrow 1218 in its asterisk context FIG. 12B. By storing the data in 1210 and storing the data to indicate that the text line node 766 is linked by an asterisk node 1214, as indicated by arrow 1220. Thus, if the data stored in the text line context node 766 indicates that the point and location of the line has changed for some reason in the electronic document 1200, the links 1218, 1220, if possible, the context node in the changed electronic document. Will allow the application to detect that the rendering of the asterisk should be adjusted for the location and / or spatial location corresponding to the rendering of the data associated with 766. Alternatively, if necessary, the parser may replace the star node 1214 (or other appropriate node associated with the star comment) or one of the individual word nodes (eg, a word node) without departing from the present invention. (768a)). Also, if desired, ink picture node 1212 may be a source node anchored to line node 766 (or other suitable node corresponding to typed text in this example). Any suitable link placement or data representing a link can be used without departing from the present invention.

The "flowchart" type of linked annotation is shown with the aid of FIGS. 13A and 13B. In this example, electronic document 1300 includes a flowchart, which may optionally be included as part of an annotation in the entire large electronic document. As shown in FIG. 13A, the tin or flowchart is surrounded or surrounded by a first ink entry 1302 (letter “A” in this example) and a second container tin 1308 surrounded or surrounded by a first container. A second ink entry 1306 that is signed. Additionally, connector tin 1310 (arrow in this example) extends from first container tin 1304 to second container tin 1308. However, it should be understood that relationships between document elements may be represented in alternative ways other than links. For example, such relationships may be represented in the structure of the data tree itself, for example, through the use of container nodes containing associated document elements.

FIG. 13B shows an example data structure 1320 that, when fully parsed, may be associated with flowchart annotations of the electronic document 1300 of FIG. 13A. In this example data structure 1320, a root node 1322, which may correspond to all or a portion of the electronic document 1300, is provided to nodes in a flowchart type annotation as parent node. The first context node for the particular annotation structures is an ink picture node 1324 corresponding to container 1304. An ink container node 1326 is provided to the ink picture node 1324 as a child node, and specific stroke (s) associated with the ink container annotation 1304 are stored in the leaf node (s) 1328, or an ink container node 1326 (or other nodes) are stored as attributes.

As mentioned above with respect to FIG. 13A, the annotation further includes electronic ink text corresponding to the letter “A” 1302. In common with the other electronic ink text described above, the electronic ink associated with this ink text is stored in a hierarchical data structure 1320 as a paragraph node 1330, which includes a line node 1332. The line node 1332 includes an ink word node 1334, the ink node word 1334 corresponding to individual strokes in the ink text annotation or one or more leaf nodes corresponding to an attribute comprising the stroke data ( 1336) optionally. The information stored in the data structure 1320 is a paragraph node (of letter " A ", as the ink container node 1326 (" source " node) of container annotation 1304 is shown by arrow 1338 in FIG. 1330) (“Destination” context node). Alternatively, if desired, the ink word node 1334 or line node 1332 can serve as a destination node. Additionally, the information stored in data structure 1320 also indicates that paragraph node 1330 is linked by ink container node 1326, as indicated by arrow 1340 in FIG. 13B. In this example, the data structure for the ink container 1308 and the letter "B" 1306 of the annotation is the same as the data structure of the ink container 1304 and the letter "A" 1302 including the same general link structure. Since the structure is shared, the various nodes in the data structure 1320 for the letter "B" 1306 and its associated container 1308 share the same reference numerals as those associated with the letter "A" and its container, but the letter " b "follows reference numerals for the ink container 1308 and the ink word 1306 in FIG. 13B. Of course, the various hierarchical arrangements and link structures or link data may differ from those previously described without particularly departing from the present invention.

The annotation further includes a connector annotation 1310 indicated by an ink picture node 1342 in the data structure 1320. As described above with respect to FIG. 11B, the data structure for the ink picture representing the ink connector annotation may further include an ink connector node 1344, which ink connector structure 1344 may include one or more attributes or leaf nodes. (S) 1346, which leaf node (s) 1346 include data associated with the particular stroke (s) that make up the connector 1310. To complete the annotation data structure of this example, the data associated with the ink connector annotation 1310 (source node) is linked with the data associated with two container annotations 1304 and 1308 (destination nodes). In particular, as shown in the example of FIG. 13B, the information indicates that the ink container node 1326 is linked to the ink connector node 1344 (indicated by arrow 1348) as the “instructions from” annotation type. To a data structure 1320. Additionally, information is stored in data structure 1320 to indicate that ink connector node 1344 is linked to ink container node 1326b (indicated by arrow 1352) as the "instructions to" annotation type. do. In this way, the data stored for the links indicates that the connector 1310 points from the circle container 1304 to the circle container 1308, as shown in FIG. 13A. Therefore, the presence of several links will allow the annotation to remain intact, even if the electronic document 1300 changes in some way, since the movement of a portion of the annotation within the electronic document 1300 precedes other portions of the annotation. Of course, the overall structure described above may be linked to some of the underlying document elements described above without departing from the invention.

Similarly, a comment may include two enclosures connected by a connector with written comments, for example, "switch these!". If a portion of the comment is moved, it will be necessary to move it independently of the other parts of the comment, such as when words are inserted between two enclosures. Links between context nodes representing electronic ink annotations and nodes representing non-ink content can be used by a software application to reposition pieces of the annotation, when the positioning operation is complete. You can tie them together again.

14A and 14B show another example of an electronic document 1400 having a flowchart type annotation or ink structure. In this example, the electronic ink annotation for the letter "A" 1402 is the letters "B" to the electronic ink connectors 1408 and 1410 between the letters "A" and "B" and "A" and "C". Connected to electronic ink annotations for 1404 and "C" 1406, respectively.

An example data structure 1420 for the electronic document 1400 after parsing is shown in FIG. 14B. Data about the entire electronic document or portions thereof may be stored at the parent root node 1422. The data corresponding to the electronic ink words "A", "B" and "C" are the same paragraph nodes 1424a, 1424b, 1424c, line nodes 1426a, 1426b, 1426c, as used in several shapes above. Ink word nodes 1428a, 1428b, 1428c, and stroke node (s) 1430a, 1430b, 1430c. Repeated descriptions of these nodes are omitted. Similarly, the ink connectors have the structure of ink picture nodes 1432a and 1432b, ink container nodes 1434a and 1434b, and stroke node (s) 1434a and 1436b previously used in FIGS. 11B and 13B, and repeated descriptions are omitted. do. Of course, changes in these specific data structures, including the possible changes described above, can be used without departing from the invention.

To complete this example annotation data structure, the data associated with the ink connector annotation 1408 is linked to the data associated with the two ink word annotations 1402 and 1404 to which it is linked, and the data associated with the ink connector annotation 1410. Is linked to the data associated with the two ink word annotations 1402 and 1406 that are connected. In particular, as shown in the example of FIG. 14B, the ink picture node 1432a (the " source " node) is indicated by the ink word nodes 1428a, 1428b (represented by arrows 1438 and 1440, respectively). " Information is stored in the data structure 1420 to indicate that. Similarly, the data structure 1420 to indicate that the ink picture node 1432b (the "source" node) points to the ink word nodes 1428a and 1428c (represented by arrows 1442 and 1444 respectively) ". The information is stored in. The ink word nodes 1428a, 1428b, 1428c serve as "destination" nodes. Because of the various links, movement of a portion of the annotation of the electronic document 1400 may precede other portions of the annotation, so that the annotation may remain intact even if the electronic document 1400 is changed in some way. In addition, such annotations may be linked with one or more other elements in an electronic document, such as typed text, pictures, images, and the like, without departing from the invention.

15A-15C illustrate example data structures including electronic ink annotation in the form of examples of electronic document 1500 and a table having a plurality of rows and columns. In particular, as shown in the example of FIG. 15A, the electronic document 1500 includes a table 1502 with two rows and two columns (consisting of four cells in total), each cell of the table It consists of electronic ink boundaries and contains electronic ink text input.

After parsing is complete, an exemplary data structure 1510 for the electronic ink table 1502 shown in FIG. 15A is shown in FIG. 15B. In this example, data structure 1510 includes a root node 1512 that may correspond to all or part of electronic document 1500. Under root node 1512, relevant data about the table is stored in table node 1514. This may include, for example, data indicative of table size, table position, number of rows, number of columns, bounding box size, and the like. Under table node 1514, separate row nodes are provided for each row of the table (in this example, two row nodes 1516, 1518 are provided, as shown in FIG. 15B). The row node of each of these exemplary data structures further includes one or more cell nodes (in the example shown, row node 1516 includes two cell nodes 1520 and 1522, and row node 1518 is Two cell nodes 1524, 1526). Since each cell of this example contains electronic ink text data, the rest of the hierarchical data structure 1510 of this example has the familiar shorting node, line node, ink word node and stroke node (s) described in detail above. Of course, the table cells are in addition to and / or electronic ink data including, for example, typed text data and / or various data types described above in detail with respect to FIG. 6I without departing from the invention. It may instead contain one or more other data types. As another alternative, instead of row nodes 1516, 1518, the data structure may include column nodes, which may include cell nodes, shorting nodes, and the like, as shown in FIG. 15B. The ink strokes that make up the actual table are, for example, portions of table node 1514 (eg, attributes associated with that node), ink picture nodes that depend on table node 1514 (associated strokes stored below it). And / or any other suitable location within data structure 1510.

An alternative example data structure 1530 for an electronic document 1500 that includes a table 1502 is shown in FIG. 15C. In this example, root node 1512; Table node 1514; Row nodes 1516 and 1518; And cell nodes 1520, 1522, 1524, 1526 are the same as shown in FIG. 15B. However, under various cell nodes 1520, 1522, 1524, 1526, rather than including electronic ink text data as child nodes, paragraph nodes 1528a, 1528b, 1528c, 1528d of electronic ink text data (or any Other suitable or desired data structures) are linked to respective cell nodes 1520, 1522, 1524, 1526, respectively, as shown by arrows 1530a, 1530b, 1530c, 1530d in FIG. 15C. The cell nodes of this example can serve as container type "source" nodes for ink words contained therein, and ink paragraph, line or word nodes serve as "destination" context nodes. Because of the various links, the movement of a portion of the table annotation in the electronic document 1500 may precede other portions of the annotation, thus keeping the table annotation intact, even if the electronic document 1500 is changed in some way. Can be linked. The table can also be linked to several other nodes that exist based on the underlying electronic document. Again, as an alternative, row nodes 1516 and 1518 may be replaced with column nodes including various cell nodes without departing from the present invention. Also, the ink strokes that make up the actual table may be placed at the table node 1514 at any location, for example, as part of the table node 1514 (eg, as attributes associated with that node) without departing from the invention. As a dependent ink picture node (with associated strokes stored below), and / or at any other suitable location within data structure 1510.

Of course, various aspects of the various exemplary annotation types described above merely describe possible annotation types, the use of annotations, and examples of information that may be included within the annotations. Those skilled in the art will appreciate that many modifications and variations can be made without departing from the present invention. For example, annotations may include many other data types (such as electronic ink text, pictures, images, typed text and images, etc.) in many other combinations and exchanges without departing from the invention.

In addition, various linking schemes are described in relation to the various data structures described above. Such link plans or techniques are merely examples of how various data nodes can be linked together. Data associated with the link or any suitable link placement can be stored without departing from the present invention. For example, it may be possible to represent each node as an individual identifier (such as a “globally unique identifier”, ie, a GUID), and using GUID pairs (or more). As another alternative, a single link may be used for the link nodes rather than the dual link arrangement described for some examples above. As another possible alternative, the node can be linked by itself if necessary or desired (eg, self-linking where the source node and the destination node are on the same node). Those skilled in the art will appreciate that any suitable manner of associating or linking various data sets with each other may be used without departing from the present invention.

<Analysis of Ink>

Various exemplary techniques for analyzing electronic ink in an electronic document in accordance with examples of the present invention will now be described. In particular, FIGS. 16A-16E illustrate a flowchart showing steps of analyzing a document in accordance with various embodiments of the present invention. 17-26 then show the relationships between the different components used during the analysis process.

Returning now to FIG. 17, this figure illustrates a software application 1701. The software application 1701 maintains a document 1703 that may include both electronic ink data 1705 and non-ink data 1707, such as typed characters or images. As discussed in detail above, the attributes of both electronic ink data 1705 and non-ink data 1707 may be described in a hierarchical data structure such as a tree. To begin analysis of the electronic ink, at step 1601, the software application 1701 creates such a data structure identified as the analysis context object 1709 in FIG.

18 shows an example tree 1801 of the type that may be included in the analysis context object 1709. The tree 1801 includes a root node 1803 and a shorting node 1805. The paragraph node 1805 corresponds to, for example, a paragraph of electronic ink text that may have been previously identified from a previous analysis process. The paragraph node 1805 is connected to two line nodes 1807, 1809, which indicate two lines in the paragraph of the electronic ink text. Line node 1807 is in turn associated with two word nodes 1811, 1813. Each word node 1811, 1813 corresponds to a word in a line of electronic ink text represented by line node 1807. Word node 1811 includes ink stroke data 1815 and ink stroke data 1817. Therefore, the stroke data 1815 and 1817 correspond to the text electronic ink strokes that make up the word represented by the word node 1811. Similarly, word node 1813 includes ink stroke data 1819 and ink stroke data 1821, which correspond to the text electronic ink strokes making up the word indicated by word node 1813. The tree 1801 also includes stroke data 1831-1827 associated with only the root node 1803. Such ink data 1823-1827 indicate new ink strokes that are unclassified or associated with another ink stroke or other document element.

Therefore, without storing the strokes in separate stroke nodes, various nodes may have an associated "strokes" attribute that stores data corresponding to the strokes associated with the node. For example: in a list item (a) unclassified context nodes may include one or more "strokes" attributes with one or more strokes that need to be analyzed; (b) the ink word nodes may include one or more "strokes" attributes that include one or more strokes that make up the ink word; (c) one or more "strokes" attributes where picture nodes include one or more strokes that make up a picture, and (d) one or more "strokes" where bullet nodes comprise one or more strokes that make up a bullet. "Attributes may be included. Of course, as alternative implementations of the present invention, individual stroke nodes can be used in the data tree to represent the individual strokes of the former, for example, rather than associating ink strokes with word nodes or picture nodes.

Although the tree 1801 shown in FIG. 18 already contains ink strokes organized into words, lines and paragraphs, the tree 1801 is associated with another ink stroke or document element or is only unclassified with only new ink strokes. It should be understood that it may include. For example, if the user initially enters electronic ink into document 1705, these initial ink strokes will not be analyzed. It should also be understood that the tree 1801 is merely exemplary and is overly simplified to facilitate understanding of various embodiments of the present invention. For example, a line node will typically be associated with a plurality of word nodes, each word node may include stroke data for some ink strokes. The tree 1801 is merely ink-related nodes, while the analysis context object 1709 represents a node representing non-ink document elements, such as images and typed text, as described in detail above. It should also be noted that these may be included.

Returning now to FIG. 17, as some examples of the present invention, the software application 1701 will create and maintain its own analysis context object 1709. For these software applications 1701, the software application 1701 may simply provide a reference to the subject 1709 for analysis. However, for such software applications 1701, the application 1701 must include the mechanisms needed to create, populate, and maintain the subject of analysis. However, some software developers will not want to raise the problem of providing the application 1701 to these mechanisms.

Thus, however, as other examples of the present invention, the software application 1701 can demonstrate another object to create an analysis context object 1709. For example, the software application 1701 can use an ink analysis tool or other object to create and / or maintain an analysis context object 1709. For example, software application 1701 may provide or identify unanalyzed ink data to an ink analysis tool. Depending on the instructions from the software application 1701, the ink analysis tool is then used to determine the entire document 1703, or, alternatively, the specific area (or the area of the document containing the unanalyzed ink identified by the software application 1701). Analysis context object 1709 representing regions) can be created. As some software applications 1701, the software application 1701 may then maintain and update the analysis context object 1709 on its own or using services provided with an ink analysis tool or another object. As other software applications 1701, the software application 1701 may not be able to hold the analysis document object 1709, and instead have a new analysis document object 1709 when it has ink analysis tools or other objects are needed. Create

Typically, the analysis context object 1709 will contain information about document elements for the entire document 1703. However, as some examples of the present invention, the analysis context object 1709 may include information about document elements within only a portion of the document 1703. That is, the analysis context object 1709 can only display document elements within the spatial area of the document that contain new or "dirty" ink or other data. If all document elements in the area of the document containing the electronic ink have already been analyzed, these pre-analyzed document elements may not be included in the analysis context object 1709. As another alternative, the application may maintain a separate analysis context object 1709 for each page or other subset of data relating to the entire electronic document 1703.

Once the software application 1701 has created an analysis context object 1709, at step 1603, the ink analysis tool 1901 may at least part of the analysis context object 1709, as shown in FIG. 19. Copy In particular, if the analysis context object 1709 is not limited to areas containing new ink and / or other data as described above, the software application 1701 includes new ink and / or other data to be analyzed. Areas of the document 1703 will be specified. The software application 1701 then causes the ink analysis tool 1901 to copy a portion of the analysis context object 1709 corresponding to the designated area of the document 1703. (Of course, if the analysis context object 1709 is limited to describing areas containing new ink, the ink analysis tool 1901 can copy the entire analysis context object 1709.)

 The application 1701 knows that the ink has not been analyzed beforehand, but any portion (s) of the electronic document 1703 (eg, pre-analyzed ink, underlying data in the electronic document, etc.) is new ink. (Which part (s) are present in the ink analysis tool 1901) is not known. Therefore, in at least some examples of the invention, the application 1701 will make large sections of the electronic document 1703 available to the ink analysis tool 1901 via the analysis context object 1709. The application 1701 leaves the ink analysis tool 1901 to determine the information actually required from the application 1701 via the analysis context object 1709.

In response to the call to analyze new data, the ink analysis tool 1901 may, if necessary, analyze contextual objects 1709 required to analyze other data within the area designated by the new ink and / or software application 1701. To get the information from the call, the callback is made several times to the analysis context object 1709. For example, in addition to new ink and / or other data, the ink analysis tool 1901 may include information about ink and / or other data within a designated area that has already been analyzed, or non-ink document elements within that designated area. The analysis context object 1709 may be queried for information about. The “designated area” corresponds to data in a spatial area at or near the area that contains new ink or other data to be analyzed, in at least some examples of the invention.

Both pre-analyzed ink and non-ink document elements, in particular those elements located near newly input data to be analyzed, may provide a context for improving the analysis of unanalyzed ink or other data. Since the ink analysis tool 1901 obtains the desired information from the analysis context object 1709, the ink analysis tool 1901 generates a document independent analysis context object 1901 with the obtained information. Therefore, the document independent analysis context object 1903 includes at least a portion of the information contained in the analysis context object 1709, but it is independent of the document 1703. The use of this "call back" technique in queries based on the spatial abstraction method described above is useful for the ink analysis tool 1901, even for large documents, to efficiently obtain and analyze the required data. Make it possible. However, it should be noted that the application 1701 can simply limit the content in the document independent analysis context object 1903 by not exposing the content to the analysis tool when requested.

Although the ink analysis tool 1901 creates a document independent analysis context object 1903, the software application 1701 should not change the analysis context object 1709 (and other threads in the application). And / or other programs may not allow to change its analysis context object 1709 during this time period. That is, the software application 1701 should not enter new data into the document 1703 during this time period and should not allow other threads and / or applications to do so. However, the process of creating a document independent analysis context object 1903 is relatively fast, which will typically not significantly affect the operation of the software application 1701. However, if desired, the results of entering data into document 1703 may be entered into the system and cached after the document independent analysis context object 1903 is created.

Once the document independent analysis context object 1903 is created, all subsequent analyzes of the unanalyzed ink and / or other data may be delayed by the software application 1701 due to the analysis of the unanalyzed ink and / or other data. It may be performed on document independent analysis context object 1901 rather than analysis context object 1709 that allows for continued normal operation without being suspended. The software application 1701 can even enter new electronic ink data 1705 into the document 1703 without interfering with the analysis of the unanalyzed ink in the document independent analysis context object 1903.

Thus, after the document independent analysis context object 1903 is created, at step 1605 the ink analysis tool 1901 creates a separate analysis thread for analyzing ink in the document independent analysis context object 1903. Thus, at step 1607, the software application 1701 can resume control of the main processing thread and continue normal operation. Then, in step 1609, the ink analysis tool 1901 provides a document independent analysis context object 1903 to the first analysis process for analysis. For example, as shown in FIG. 20, the ink analysis tool 1901 may pass a document independent analysis context object 1903 to a parsing process 2001 for parsing using an analysis thread. The parsing process additionally includes ink and / or other data such as ink text, ink pictures, tables, charts, graphics, images, music, mathematics, (resistance, capacitors, etc.) without departing from the present invention. One or more classification processes (such as schematics) to classify into various other types, such as pictures with specific symbols. It should be noted that as some examples of the present invention, the ink analysis tool 1901 can make a copy of the document independent analysis context object 1903. As will be described in more detail below, a copy of the original form of the document independent analysis context object 1903 can be used to adjust the analysis results of the analysis process to the current state of the document 1703.

After the first analysis process analyzes the ink and / or other data in the document independent analysis context object 1903, the first analysis process returns the analysis results to the ink analysis tool 1901, which in turn is the software of step 1611. Inform the application of the results of the analysis. For example, as shown in FIG. 21, if the first analysis process is a parsing process 2001, the parsing process returns the parsing results 2101 to the ink analysis tool 1901, and then (eg, For example, by firing an event, a reference to the analysis results 2101 can be passed to the software application 1701.

As will be described in more detail below, as some examples of the invention, the analysis results 2101 may be the document independent analysis context object 1903 that originally existed in the analysis process, but generated by the first analysis process. Can be changed to include new information (eg, parsing results in this example). However, as still other examples of the invention, the analysis results 2101 can be a modified copy of the document independent analysis context object 1903. By including new information generated by the first analysis process in a copy of the document independent analysis context object 1903, the original document independent analysis context object 1903 may be, for example, by a reconciler and / or It can be preserved for use in the other analytical processes mentioned above.

FIG. 22 shows an example of how a parsing process, such as parsing process 2001, may modify data tree 1801 to show layout changes generated by the parsing operation. As shown in this figure, the parsing process 2001 determines that the unanalyzed ink stroke indicated by the stroke data 1823 is part of the word indicated by the word node 1813. Thus, the parsing process disassociates the root node 1803 from its stroke data 1823 and associates it with the word node 1813 instead.

The parsing process also determines that the ink strokes 1825 and 1827 represented by the unclassified stroke data are part of the new word in the line indicated by the line node 1809 that is not previously identified. Thus, parsing process 2001 generates a new word node 2201 and associates the line node 1809 with the word node 2201. The parsing process 2001 then associates the stroke data 1825, 1827 with the new word node 2201. Therefore, the parsing processes 2101 describe the relationships between the unanalyzed ink strokes and other ink strokes (or other document elements) identified by the parsing process 2001 in advance. Further, in addition to showing the relationship changes determined by parsing process 2001, the parsing results 2101 may also include classification information determined by the parsing results 2101. For example, each case of ink stroke data 1823-1827 can be modified to classify the corresponding ink stroke as text ink stroke rather than picture ink stroke. In at least some examples of the present invention, the parsing results 2101 can be displayed in the document independent analysis context object 1903 as needed.

As mentioned above, while the first analysis process analyzes the document independent analysis context object 1903, the software application 1701 includes both new ink data 1705 and new non-ink data 1707. New data can be entered into the document 1703. Thus, the results of the first analysis process may no longer be applicable to the current state of document 1703. For example, the parsing process may determine that the ink stroke is associated with the word, but the user may delete the ink stroke entirely from the document 1703 (eg, according to the entire associated word or more). Thus, the analysis results 2101 should be adjusted to the current state of the document 1703 at step 1613. An exemplary adjustment process is shown generically in FIG. 23, which will be described in more detail below.

As some examples of the invention, the software application 1701 can manually adjust its analysis results 2101 to the current state of the document 1703. That is, the software application 1701 can sift through the analysis results 2101 to determine that the results relate to document elements in the current document 1703. However, as other examples of the present invention, the ink analysis tool 1901 may include the analysis results 2101 as an analysis context object 1709 that reflects the current analysis context object 1709 (ie, the current state of the document 1703). You can provide adjustments to adjust with)). As will be discussed in more detail below, the adjustment function of the various embodiments of the present invention is based on the conflict between the analysis results 2101 from the first analysis process and the analysis context object 1709 for the current state of the document 1703 ( You will identify a conflict or "collision". The ink analysis tool 1901 then updates the analysis context object 1709 based on the analysis results in step 1615, and a new document independent analysis from the adjusted analysis results of the analysis context object 1709 in step 1617. It will create a context target 1903.

While the ink analysis tool 1901 adjusts its analysis results 2101 to the current analysis context target 1709, the software application 1701 should not change the analysis context target 1709 (and the analysis context target). (1709) should not be allowed to be changed by the application and / or other threads in any other application. That is, the software application 1701 should not enter new data into the document 1703 and / or allow other threads and / or applications to enter the new data into the document 1703 until the adjustment is complete. Should not. As with the process of initially creating the document independent analysis context object 1903, its coordination process is relatively fast and will typically not significantly affect the operation of the software application 1701. However, if necessary, the attempted user input can be input and cached after the adjustment process is completed.

After the ink analysis tool 1901 adjusts the analysis results 2101 to the analysis context object 1709 for the current state of the document 1703, the ink analysis tool 19901 selectively modifies the adjusted analysis results. Can be provided to a second analysis process for analysis. For example, as shown in FIG. 24, the ink analysis tool 1901 can hand the adjusted parsing results 2401 (eg, text, music, mathematical information or other specific data types) for recognition. To the ink recognition process 2003 (for writing recognition). In particular, the ink analysis tool 1901 will recreate a separate analysis thread for performing the second analysis process in step 1619. Since the ink analysis tool 1901 creates such a separate thread for executing the second analysis process, the software application 1701 can resume normal operation in step 1621, and even new electronic ink data 1705. It can accommodate new input data comprising a.

Next, at step 1623, the ink analysis tool 1901 provides the adjusted results 2401 from the first analysis process to the second analysis process. Once the second analysis process has begun, the application 1701 can begin another execution of the first analysis process for the new unanalyzed ink. Therefore, multiple analysis processes can be executed simultaneously, so that while the second analysis process operates with the results of the first analysis process, the application 1701 performs the first analysis process to prepare for the next set of analysis results. You can restart it.

Advantageously, performing the analysis processes in parallel can improve the operation of the software application 1701. For example, the parsing analysis process is typically fast relative to the recognition analysis process, and the parsing analysis results can be used to implement an accurate selection operation and a space-insertion operation without using the recognition analysis results. have. Therefore, because other analysis processes can be performed in parallel, the new ink added to the document 1703 may wait for the recognition analysis process to finish the old ink data before it has space to be selected or correctly inserted therein. no need. In addition, various embodiments of the present invention provide asynchronous ink analysis to isolate not only parser and recognizer algorithm developers, but also application developers from multithreading problems that improve maintainability and simplify development processes for both groups. And allow changes to interact between analysis processes without making changes to the application.

As mentioned above, the ink analysis tool 1901 is updated to include relevant portions of the analysis results 2101 and then adjusted by creating a new document independent analysis context object 1903 from the analysis document object 1703. Analyzed results can be provided. However, as other examples of the invention, the ink analysis tool 1901 may simply update the original document independent analysis context object 1903 with its analysis results 2101 to reflect the current state of the document 1703. Can be. In addition, the ink analysis tool 1901 can simply provide relevant ink data to be analyzed differently from the second analysis process.

Therefore, various embodiments of the present invention adjust the results of the previous analysis process to the current state of the document 1703 before executing the next analysis process. However, it should be understood that this intermediate adjustment step may be omitted in some examples of the invention. In particular, as some examples of the invention, the results of earlier analytical processing may be provided directly to the next analytical processing without adjusting the results to the current state of document 1703. For example, if the ink analyzer process is provided separately from the ink layout process, ink classified as the handwritten text can be provided to the ink layout analyzer process without an intermediate adjustment step.

In another arrangement, after the second analysis process 2003 has finished analyzing the adjusted results 2401 from the first analysis process, the second analysis process 2003 may then perform its analysis as shown in FIG. The results 2501 are returned to the ink analysis tool 1901. The ink analysis tool 1901 then informs the software application 1701 of the results 2501 of the second analysis in step 1625.

As mentioned above, while the second analysis process analyzes the adjusted results 2401 of the first analysis process, the software application 1701 is responsible for documenting new data including the new electronic ink data 1705. 1703). Therefore, the second analysis processing results 2501 are no longer applicable to the current state of the document 1703. Thus, the results 2501 of the second analysis process 2003 should also be adjusted to the current state of the document 1703 at step 1627. An exemplary adjustment process is shown comprehensively in FIG. 26, which will be described in more detail below.

Again, as some examples of the present invention, the software application 1701 can manually adjust the second analysis results 2501 to the current state of the document 1703. Alternatively, as other examples of the present invention, the ink analysis tool 1901 may provide an adjustment function to adjust the analysis results 2101 to the current analysis context object 1709. This adjustment function will identify an conflict or “collision” between the analysis results 2501 from the second analysis process and the analysis context object 1709 for the current state of the document 1703, and then the ink analysis tool ( 1901 will update the analysis context object 1709 based on the analysis results at step 1629. This process may then be repeated as needed to analyze the new ink input or ink during the analysis processes.

From the foregoing discussion, ink analysis techniques in accordance with various embodiments of the present invention allow electronic ink to be analyzed asynchronously from software application 1701, where the software application 1701 includes input data that contains new electronic ink. Continue to be accepted during the analysis process. Although the above discussion describes only two analytical processes, it should be noted that any number of analytical processes may be performed in parallel at the same time. For example, various implementations of the invention may utilize a first parsing process, a second parsing process, and a recognition process. In this way, once each analysis process is completed, the results can be transferred to the next analysis process, and the first analysis process can again be executed simultaneously and in parallel to the newly input ink.

In addition, as various embodiments of the present invention, one or more analysis processes may be executed sequentially. For example, with some software applications, a user can specify a specific language for an ink stroke or collection of strokes, such as English or Japanese. This language designation may then be included in the analysis context object 1709 and the document independent analysis context object 1903. Based on the language designation, the ink analysis tool 1901 routes the electronic ink designated in the first language to a recognition process tailored to the first language, and then routes the electronic ink designated in the second language to the recognition process. Routing to other recognition processes tailored to the second language. In this same manner, input data classified and / or designated in other specific types may be sent to other specific identifiers such as musical identifiers, electrical symbols, mathematical symbols, flowchart shapes, graphic shapes, and the like.

In addition, since the analysis processes need not be part of the software application 1701, any desired analysis process can be used to analyze the electronic ink in the software application 1701. For example, a software application developer may develop and use a parsing process or recognition process that is particularly suitable for analyzing the expected electronic ink input for software application 1701.

Still other variations of these techniques can be used by various implementations of the invention. For example, in some situations, it may not be desirable for the software application 1701 to maintain an internal document tree that reflects relationships between various document elements in the document 1703. Instead, the software application 1701 may be intended to simply use simple state information regarding the electronic ink input. As such software applications 1701, the software application 1701 may omit generating an analysis context object 1709 that reflects the current state of the premise document 1703. Rather, the software application 1701 can create a particular-purpose analysis context object 1709 that only includes information about a particular ink input that the software application 1701 wants to be analyzed. By using the specific-purpose analysis context object 1709, while using the desired analysis processes to analyze the ink in the document, the software application 1701 maintains an internal data structure corresponding to the state of the document 1703. Complexity can be avoided.

In addition, while the foregoing description discusses asynchronous analysis of electronic ink, it should be noted that various embodiments of the present invention may allow the ink analysis tool 1901 to simultaneously analyze the ink. For example, some implementations of the invention may allow software application 1701 to perform an immediate and simultaneous analysis of electronic ink. In addition, simultaneous analysis may be for ink only or for the entire document 1703 within a particular area of the document 1703.

Ink Analysis Tool

As discussed above, the ink analysis tool 1901 performs various functions to facilitate the processing of electronic ink. For example, in accordance with various implementations of the invention, the ink analysis tool 1901 receives information from a software application 1701 that specifies one or more regions within the document 1703 from which ink data is to be analyzed, and the regions. The software application 1701 is queried to obtain information about the fields, and then, based on the information, a document independent analysis context object 1903 is constructed. The ink analysis tool 1901 also provides a document independent analysis context object 1903 to one or more analysis processes for analysis, and then adjusts the results of the analysis process to the current state of the document 1703. The various processes may be performed as detach processes, through a thread pool, through detached machines in a cluster or any other suitable or desired way using detach threads (as described above).

According to other examples of the invention, various software objects and tools may be used to implement the ink analysis tool 1901 and perform these functions. However, as some examples of the present invention, the ink analysis tool 1901 may be implemented as an application programming interface (API). In particular, according to some examples of the invention, the ink analysis tool 1901 is a group of software subject routines and associated information that can be called by the software application 1701 required to analyze the ink in the document 1703. Can be implemented.

As will be appreciated by those skilled in the art, an application programming interface may be organized into categories or "classes" belonging to software objects. A "target" is a collection of storeable state values and executable actions. In particular, an object may maintain other types of state values referred to as "fields" and "properties", which may be associated with or retrieved by that object. For external software applications or another software object, these attributes are read only (identified by the use of the term "{get;}"), write only (term "{set;} "Identified by the user of", or both readable and writable (identified as "{get; set}"). A subject may also often execute tasks or “methods” in a request or “call” of a software application or another subject. Therefore, the application programming interface may include various objects that can be commanded by another software application to perform specific tasks or to provide specific information on request.

It should be noted that various embodiments of the present invention may be implemented to require that the software application 1701 provide only the minimum amount of information needed to complete the desired analysis process. For this reason, some attributes, which will be described in detail below, may be discussed as read only (ie "{get;}") or write only (ie "{set;}") attributes, but various alternatives of the invention It may be both readable and writable for implementations. For example, an attribute that defines "hints" to be used by the analysis process to analyze the electronic ink may be identified as a read-only attribute for situations in which the software application 1701 should not be intended to use the hints. have. This allows objects created by the ink analyzer to read at least these attributes, and that the value of these attributes is a null default value (i.e. any "hints") by the software application 1701. Not desired). However, if the software application 1701 wants to use hints for ink analysis, this attribute is read to allow the software application 1701 to change the value of this attribute from zero to specify the desired hints. It will be possible and recordable.

According to various embodiments of the present invention, an API (hereinafter referred to as an ink analysis API) that specifies an implementation of the ink analysis tool 1901 may include two core classes. The first class will be referred to as the "analysis context" class, and the second class is the "Ink Analyzer" class. Components of the "analysis context" class are used to create an analysis context object 1709. Thus, as various implementations of the invention in which the software application 1701 creates and maintains its own analysis context object 1701, one or more components of this class may be omitted from the ink analysis tool. Instead, one or more of these components may be implemented by the software application 1701 itself.

The components of the "Ink Analyzer" class are then used to create and use objects that provide the document independent analysis context object 1903 to the analysis process, determine when the results of the analysis are generated, and document the results of the analysis. 1703) to the current state. These and other classes that may be included in the ink analysis API in accordance with various embodiments of the present invention will be described in detail below.

Returning to the analysis context class first, this class is implemented by the host application 1701 to create the analysis context object 1703, which serves as a proxy view on the internal document tree of the software application 1701. Do it. Alternatively, as mentioned above, if the application implements its own analysis context object, one or more components in this class may be omitted from the ink analysis tool 1901. As described above, the analysis context object 1703 includes all unanalyzed ink data, and the analysis context object 1703 is used to identify that the unanalyzed ink data should be analyzed. The analysis context object 1703 also includes information about the ink that has already been analyzed. This already analyzed ink can now be used to determine how the unanalyzed ink should be analyzed, and itself can be changed during the analysis of the unanalyzed ink. In addition, the analysis content object 1703 includes information about non-ink content of the document 1703, which is used to properly classify ink as annotations for non-ink content.

According to various embodiments of the present invention, the analysis context class includes a constructor that, when invoked by the software application 1701, creates the analysis context object 1709. This class may also include various attributes for the analysis context object 1709 that include an attribute named "Dirty Region {get;}". The dirty area attribute defines the portion of the document (and portion of the analysis context object 1709) that contains the unanalyzed ink data. In various embodiments of the present invention, the dirty area may define a disjoint area. The dirty region is specified as an AnalysisRegion, which will be described in more detail below, but can simply be a collection of rectangular regions within the document. If the dirty area is empty, no analysis will occur. This class further includes an attribute for the analysis context object named "Margins {get;}", which declares a page area within the document 1703, which is considered margins. For example, an analytical process that analyzes the layout and determines the classification of the electronic ink can use these attributes to help determine the classification of the electronic ink that is non-ink content (eg, margin ink).

In addition, the analysis context class may include an attribute named "Rootnode {get;}", which identifies the best or root context node in the analysis context target 1709. As discussed in more detail above, this root node includes all other context nodes as child context nodes for a given analysis context object 1709. It should be noted that as various embodiments of the present invention, the root context node should be of that context node type "Root". As examples of the invention in which the application 1701 implements its own analysis context object 1709, the analysis context object 1709 may have other context nodes, such as siblings of the root context node. It should also be noted that the components of the Ink Analyzer class may be limited to considering the context nodes contained by the root context node.

The analysis context class may additionally include an attribute "Analysis Hints {get;}", which returns an array of analysis hint objects set by the software application 1701. As discussed in more detail below, the analysis hint subjects can include any type of information that can assist the analysis process. Such information may include, for example, as factoids, guides, or word lists. This also includes information that sets the language to be used for analysis, such as specifying unanalyzed ink as only handwritten text or just a picture, or identifying the ink as a list, table, shape, flowchart, connector, container, etc. Contains information that provides derivation of any kind of.

In addition to these attributes, the analysis context class may also include various methods that may be invoked by the software application 1701 having, for example, an analysis context target 1709 that performs a task. For example, the analysis context class may include a method named "FindNode (Guid id)". Each node of the analysis context target 1709 has a globally unique identifier (ie, GUID), and this method will be located at the node specified by the call somewhere within the analysis context target 1709. Since the method can be called from many time consuming operations, this lookup method should be implemented as efficiently as possible.

Like the analysis context class, the Ink Analyzer class also defines a public constructor that, along with various properties, allows the software application 1701 to create a case of that class (ie, an Ink Analyzer target). . For example, it may include an attribute named "User Interface Context {get; set;}" which defines a processing thread in which the results of the analysis process are returned. This property allows the results to be synchronized to another object. For example, if this is set to the main form, parser results will be fired to the application main thread. It may also contain the attribute "AnalysisOptions AnalysisOptions {get; set '}", which specifies various criteria that can be used in the analysis process. Such criteria may include, for example, enabling text recognition, enabling the use of tables, enabling the use of lists, enabling the use of annotations, and enabling the use of connectors and containers.

The Ink Analyzer class will contain several methods. For example, such a class might contain a method named "AnalysisRegion Analyze ()". This method starts the concurrency analysis process. Document element data is communicated to this method describing the current state of the document 1703 and indicating which ink in the document 1703 needs to be analyzed. In some implementations of the invention, the document element data may be provided as an analysis context object 1709 (ie, AnalysisRegion Analyze (Analysis Context)), as mentioned above. Alternatively, individual ink strokes may use references to those strokes (ie, AnalysisRegion Analyze (Strokes)) or Ink Analyzer objects (eg, InkAnalyzer.Strokes {get; set}), which can be passed to an analysis process, referred to as an attribute.

Once the analysis process is complete, this method will return the reference to the document independent analysis context object that has been modified to include the results of the analysis process. The method also returns an AnalysisRegion value describing the area in the document from which the results were calculated.

The Ink Analyzer class can also include a method named "AnalysisRegion Analyze (AnalsisContext, waitRegion)". This method is identical to the concurrency Analysis Region Analyze () method described above, but can only have the analyzed ink if results are required for a particular waitRegion region. In particular, a call to this method will identify the analysis context object 1709 for that document 1703 and the area of that analysis context object 1709 (referred to as "waitRegion") that the analysis process should analyze at the same time. . In various embodiments of the present invention, all other areas 1709 of the analysis context object 1709 are ignored unless the analysis process needs to analyze the content in those areas in order for the analysis process to complete the analysis of waitRegion. Will be. As discussed above, the analysis context object 1709 delivered in this way includes an attribute referred to as "DirtyRegion" that describes the areas of the document 1703 that require analysis. By specifying a particular waitRegion, the software application 1701 can obtain analysis results more quickly for one particular area of interest than having all the ink data in the analyzed document 1703.

If any of these Analyze methods are called, all available analytical processing will be executed. Also, because these Analyze methods are concurrency calls, the coordination process does not need to be executed at the end and will not execute the event fire upon completion.

The Ink Analyzer class also includes a method named "BackgroundAnalyze (AnalysisContext)". This method starts the specified analysis operation but starts a separate background analysis thread. Therefore, this method will return control to its main processing thread almost immediately after the actual analysis operation completes in the background. In particular, this method will return a value of "true" if the analysis process has been started successfully. Again, the AnalysisContext value passed in that way identifies the analysis context object 1709 for the document 1703 and indicates which ink in the document 1703 needs to be analyzed. Once the analysis operation completes the background thread, the Results event is raised to allow the software application 1701 to access the results. The event includes the results and the adjustment method used to integrate the results back to the analysis context target 1709 for the current state of the document 1703 when the results are returned.

Note that each of these three Analyze methods in turn calls the method "Clone" in the "Analysis Region" class, which will be discussed in detail below. Using the "Clone" method, these Analyze methods create an independent document analysis context object that is then modified by the analysis process to show the analysis results.

The Ink Analyzer class may also include a method named "Reconcile (AnalysisContext current, AnalysisResultsEventArgs resultArgs)" that the software application 1701 calls after receiving a result event caused by calling the BackgroundAnalyze (AnalysisContext) method. The Reconcile method compares the analysis results included in the document independent analysis context object with the current version of the analysis context object 1709 maintained by the software application 1701. The method identifies nodes that need to be added and removed from the current version of the analysis context object 1709, and identifies the following attributes of the existing node: its recognition results, its location, ink strokes associated with the node or analysis operation. Identifies which of any other data associated with the results of. The method also records these identified changes in the current version of the analysis context object 1709. This method is sensitive to context node order, such as the order of word context nodes on a line context node.

The results of the analysis (i.e. the value of the attribute AnalysisResultsEventArgs) are passed back in this way because they contain public results structure and private results structure. The personal structure is included to prevent the software application 1701 from changing any of the analysis results before the adjustment process.

The Ink Analyzer class may also include methods named "Recognizers RecognizersPriority ()" and "SetHighestPriorityRecognizer (recognizer)". If the ink needs to be recognized, the appropriate recognizer will be used based on the languages and abilities. Thus, the Recognizers RecognizerPriority () method returns the recognition processes in the order in which the recognition processes will be evaluated by the Ink Analyzer object. The order is determined per system depending on the available recognition processes, but can be ignored for the software application 1701 by calling SetHighestPriorityRecognizer (recognizer) on the Ink Analyzer object. The InkAnalyzer will be listed through this ordering list until a suitable identifier can be found. The SetHighestPriorityRecognizer (recognizer) method raises the priority of recognition processing. By raising the priority of a particular recognition process, that recognition process will be used if it conforms to the required language and capabilities of the current recognition operation. In essence, the SetHighestPriorityRecognizer (recognizer) pushes the specified recognition process to the top of the list returned by the RecognizersPriority method.

The Ink Analyzer class also includes a method named "AnalysisRegion Abort ()", which can take an analysis context object as a parameter. This method allows the foreground or background analysis operation to terminate early. This method returns an analysis region describing the analyzed region prior to abort. Therefore, if the software application 1701 intends to continue the analysis operation at a later time, this area may be merged into the DirtyRegion of the analysis context target 1709 for the current state of the document 1703. In addition, the Ink Analyzer class may include an event named "AnalysisResultsEventHandler" that actually fires the InkAnalyzer target frequently. In particular, such an event may fire at least once every 5 seconds between analysis processes. This event can be used to provide the update application 1701 with respect to the status of the ongoing asynchronous analysis process (or processes).

The Ink Analysis API can also include classes added to the Analysis Context class and the Ink Analyzer class. For example, the Ink Analysis API according to various embodiments of the present invention may include a Context Node class. This class may include various components relating to the analysis context node 1709 and the context nodes that make up the document independent analysis context object, such as an attribute named "ContextNodeType Type {get;}". As will be appreciated by those skilled in the art, each context node has a certain type, and there is a specific set of rules that each type must adhere to. This includes rules such as, for example, what types of child context nodes are allowed, and whether the strokes can be directly associated with that context node or only through the child context nodes.

Possible types of context nodes can be defined in the ContextNodeTypes enumeration, for example, the following types: InkAnnotation node representing ink data annotating non-text data; An InkDrawing node representing ink data forming a picture; An InkWord node representing ink data forming a word, a Line node including one or more InkWord nodes and / or TextWord nodes for words forming a line of text; A ListItem node, which may include nodes such as expected paragraphs, images, and the like within the list; And a List node comprising one or more ListItem nodes, each of which describes an entry in the list. The node types also include NonInkDrawing nodes that represent non-ink picture images; An Object node representing data not covered by other values of the ContextNodeType enumeration; A Paragraph node comprising one or more Line nodes corresponding to lines forming a short; A Picture or Image node representing a picture image; A Root node that serves as the best node in the analysis context object; A Table node including nodes representing items constituting a table; A TextBox node representing a text box; TextWord node; And an UnclassifiedInk node that still corresponds to unclassified ink data. The node types also include a Group node for groups of other nodes, an InkBullet node for bullet items, a Row node for document elements presented in a row of a table, and a Cell node for document elements presented in a cell of a table. It may include.

The Context Node class can also include an attribute named "GUID Id {get;}", which is the global single identifier for the current context node. In order to allow access to any desired context node, each context node within a single analysis context object must always have a single identifier. This class may also contain an attribute named "AnalysisRegion Location {get;}" that identifies the location in the document space where the relevant context node is actually located. As mentioned earlier, AnalysisRegion is a two-dimensional structure that groups together one or more possible disassembly rectangle structures. This class may also include an attribute named "StrokeCollection Strokes {get;}", which identifies ink strokes associated with the associated context node. In various embodiments of the present invention, only leaf context nodes (such as Word, Drawing, and Bullet nodes) are allowed by the Ink Analysis API to have strokes. The software application 1701 can use these attributes to refer to its strokes at the leaf node level by all ancestor context nodes (eg, the root node associates a stroke reference to all the strokes). In the analysis context object).

In addition, this class may include an attribute named "ContextNode ParentNode {get;}" that identifies the parent context node that contains the associated context node. In various embodiments of the present invention, context nodes are always created for the Root context node to depend on from the parent context node, which is a static number of analysis context objects. This class may also contain an attribute "ContextNode [] SubNodes {get;}" that identifies all context nodes that are direct children of the associated context node. In other words, this attribute will only identify child context nodes one level below the analysis context object. For example, the value of an attribute for such a Paragraph context node will only identify the line context nodes included by the Paragraph node, and not the word context nodes that are children of the line context node.

This class is also an attribute named "RecognitionResult RecognitionResult {get;}", which provides recognition results calculated by the relevant recognition analysis process or processes, because RecognitionResult can represent one or more lines of text in one or more languages. It may include. The Recognition Result is set by the Recognition Analysis process and although the RecognitionData attribute (described in more detail below) used to create the RecognitionResult object may be set to only one level of the context node tree to avoid duplication of data. For example, all context nodes within the document independent analysis context object are available. If the node does not have RecognitionData associated with it, it will either merge the recognition results of all its subnodes or extract the recognition results from its parent. This class may also contain an attribute named "Stream RecognitionData {get; set;}", which is a persistent form of RecognitionResult value. Again, the recognition analysis process generates a Stream RecognitionData value set to the associated context node. The RecognitionResult object is then constructed based on these values.

The Context Node class may further include an attribute named "ContextLink [] Links {get;}", which provides an array of ContextLink objects. The ContextLink target describes an alternative relationship between two context nodes. Although context nodes typically have sub-child relationships with other context nodes, ContextLink allows for an alternative relationship between context nodes. For example, a ContextLink may be a connection between two context nodes, anchoring one context node to another context node, suppressing one context node by another context node, or a desired link defined by software application 1701. You can allow types. ContextLinks can be added to this array by calling the AddLink method, discussed in more detail below. Similarly, ContextLinks can be removed from this array by calling the DeleteLink method, discussed in more detail below.

In addition, this class may include attributes "IsContainer {get;}" and "IsInkLeaf {get;}". The property IsContainer {get;} has a value of "true" if the related context node is not a leaf context node (ie, if the related context node contains child context nodes and thus container context nodes are considered), otherwise it is "false". (false) "value. The IsInkLeaf {get;} attribute has a value of "true" if the current context node is not a container context node, and a value of "false" otherwise. In other words, if the current context node does not contain any child context nodes, it is considered a leaf context node. In various embodiments of the present invention, while the InkLeaf context node is expected to include references in the stroke data, container context nodes do not have this limitation. Container context nodes may or may not refer to the stroke data as specified by software application 1701.

The Context Node class may also include the attribute "Rect RotatedBoundingBox {get; set;}". The value of this attribute is calculated by the layout and classification analysis process. If the ink data associated with the relevant context node is written at an angle, the bounds for that context node will still be horizontally aligned. However, the value of the RotateBoundingBox attribute will be aligned at the angle at which the ink data associated with the associated context node was written. In addition, this class may include an attribute "ReClassifiable {get;}", which informs its InkAnalyzer of the relevant context node if allowed to change the values.

In addition to these attributes, the Context Node class can also include various methods. For example, this class includes a method named "ContextNode CreateSubNode (ContextNodeType type)". This method allows the creation of child context nodes of a particular type. As various embodiments of the present invention, this method may only allow valid child types of the associated context node to be created, thus preventing malformed data structures from being created. For example, this method can only allow the Line context node to create InkWord and TextWord context nodes. This class may also include a method named "void DeleteSubNode (ContextNode node)", which deletes the referenced child context node from the associated analysis context object. However, as various embodiments of the present invention, it should be noted that this method fails if the referenced context node includes strokes or child context nodes. Also, if the reference context node is not a direct child of the associated context node, this method fails. If the software application 1701 implements its own analysis context object 1709 and in turn uses this method, a non-empty context node to prevent malformed data structures in the analysis context object 1709. Note that you do not delete context nodes that are not direct children of these or related context nodes.

This class may also include the method "ContextNode [] HitTestSubNodes (AnalysisRegion region)", which returns an array of context nodes located within that particular region. Note, however, that only direct child nodes of this element are returned, not all descendants. The region is defined by the AnalysisRegion object, which can be a collection of one or more rectangles, as mentioned above. According to various embodiments of the present invention, if any portion of the location of a context node crosses a particular area, that context node will be returned to the array. This method is used, for example, to create a document independent analysis context object and to adjust its analysis results to an analysis context object corresponding to the current state of document 1703. Therefore, this method is frequently called and should be optimized for quick repeated access by the InkAnalyzer object.

The Context Node class also includes a method named "MoveStroke (Stroke stroke, ContextNode destination)". This method moves the concatenation of the strokes from one leaf context node to another. As various embodiments of the present invention, this method is only used between leaf context nodes. It may also include a method named "MoveSubNodeToPosition (int OldIndex, int NewIndex)", which reorders the relevant context nodes for sibling context nodes. For example, if document 1703 has three words on a line, for example word 1, word 2, word 3, their order is related by an array of subnodes returned from the parent context node. . This method allows their order to be changed, so that for the relevant context node, word 1 is specified to be the last word on that line by moving the context node for word 1 from position 1 to position 3.

This class may also include a method named "AddLink (ContextLink link)", which adds a new ContextLink object to the current context node. In various embodiments of the present invention, the ContextLink object must include a reference to the associated context node in the order of the ContextLink to be successfully added to the array of ContextLinks associated with the associated context node. It may also include a method named "DeleteLink (ContextLink link)". This method deletes or removes that particular ContextLink object from the array of ContextLinks for the relevant context node. In various embodiments of the present invention, the method always successfully completes the call, even if the ContextLink is not present in the array of ContextLinks associated with the associated context node.

The Ink Analysis API also includes an Analysis Hint class. As many of the classes described above, the Analysis Hint class includes a constructor named "AnalysisHint ()", which initializes the Analysis Hint object to an empty state. This class can also contain many properties, including a property named "AnalysisRegion Location {get;}". This property specifies the location in the document 1703 (as AnalysisRegion) to which the AnalysisHint is applicable. For example, if the document 1703 is a free-form note with a title portion at the top of the page, the application 1701 may apply to that title area to specify that horizontal lines of ink are expected within that area. You can set AnalysisHint for. This Analysis Hint will help to increase the accuracy of the analysis process.

This class may also contain an attribute named "string Factoid {get; set;}" that specifies the particular "factoid" to be used for the location in the document 1703 where AnalysisHint is applicable. As is known to those skilled in the art, factoroids provide hints in the recognition process regarding the expected use of ink data (eg, regular text, numbers, zip codes, filenames and web URLs). . This class is called the "RecognizerGuide Guide" {get; set;} "and" OverrideLanguageId {get; may contain attributes named set;}. The Recognizer Guide Guide {get; set;} attribute specifies a recording guide to be applied to a location in the document 1703 to which AnalysisHint is applicable. For example, recording the guide helps the user to improve the accuracy of the recognizer analysis process by specifying and notifying the user of a recognizer analysis process to record lines or characters. Its OverrideLanguageId {get; set;} attribute specifies a Language Hint for it in the document 1703 to which AnalysisHint is applicable. Setting the Language Hint causes the InkAnalyzer target to use the specified language instead of the language specified on the context node.

This class may also include an attribute named "PrefixText {get; set;}" that specifies the text written or typed prior to the line of ink to be recognized. This class also contains an attribute named "RecognitionModes RecognitionFlags {get; set;}", which specifies the specific types of modes that the recognition process should relate to a location in the document 1703 to which the AnalysisHint is applicable. This class also specifies an attribute named "SuffixText {get; set;}" that specifies the text written or typed after the line of the link to be recognized and a specific set of words that should be used by the recognition analysis process. It may contain an attribute named "WordList WordList {get; set;}. Before the user has actually recorded input data, such as a list of medical terms that are expected to be written inside a medical form, If the expected recognition results are known, a word list can be used.

This class may also contain an attribute named "WordMode {get; set;}". If this value is "true", the analysis process will bias itself to return a single word for the entire analysis area. Also, if "true", it contains an attribute named "Coerce {get; set;}" that will force the analysis process to limit the result to any factoroid or wordlist set in the relevant hint. can do. This class may also contain an attribute named "AllowPartialDictionaryTerms {get; set;}". If this attribute value is "true", the recognition analysis process will be allowed to return partial words from the recognition dictionary.

According to various embodiments of the present invention, the Ink Analysis API may further include an Analysis Region class. Such a class may, for example, contain multiple constructors for constructing an AnalysisRegion target. For example, a first constructor for constructing an AnalysisRegion object having an area, a second constructor for constructing an AnalysisRegion object based on parameters for a two-dimensional rectangle, and an AnalysisRegion object based on spatial coordinates. It may include a third constructor constituting. For example, the default constructor could create an empty area. This class can also contain many attributes. For example, this class contains a property named "Rectangle Bounds {get;}" that receives a bounding rectangle for AnalysisRegion, and "IsEmpty {get;}" indicating whether the associated AnalysisRegion target has an empty interior. And an attribute named "IsInfinite {get;}" indicating whether or not an attribute named as is related to and its associated AnalysisRegion is set to infinite.

This class also contains a number of methods, such as the method named "AnalysisRegion Clone ()", which duplicates the associated AnalysisRegion target. This class can also include a method named "Equals (AnalysisRegion otherRegion)", which tests whether the specified AnalysisRegion target (referred to as otherRegion) is the same as the related AnalysisRegion target. This method returns a value of "true" if the interior of the specified Analysis Region object is the same as the interior of the related Analysis Region object; otherwise, it returns a value of "false".

This class may further include the method "Intersect (AnalysisRegion regionToIntersect)", which crops down the relevant AnalysisRegion target for the specified analysis region. Therefore, the resulting AnalysisRegion object will only contain regions that have overlapped or traversed the specified analysis region. These classes can also include a method named "Intersect (Rectangle rectangle)", which crops down the associated AnalysisRegion target for a given rectangle. Again, the resulting AnalysisRegion object will only contain regions that overlapped or traversed the specified rectangle. It may also include a method named "MakeEmpty ()" that initializes its associated AnalysisRegion target to an empty interior, and a method named "MakeInfinite ()" that sets an area occupied indefinitely by the associated AnalysisRegion. It is named "Union (AnalysisRegion regionToUnion)", which specifies to join or add an AnalysisRegion target to its associated AnalysisRegion target, and a method named "Union (Rectangle rectangle)", which joins the specified rectangle to its associated AnalysisRegion target. The method may further include various methods for combining or separating differently defined regions. In this way, the rectangle can be specified by the coordinate space for the associated AnalysisRegion object. Of course, this class may include a number of different ways to combine regions or extract one region from another region based on any desired definition of the regions.

The Ink Analysis API can also have a Recognition Result class. As with many of the classes described above, the Recognition Result class may contain one or more constructors. For example, such a class may contain a constructor named "RecognitionResult (Stream lattice)", which constitutes a RecognitionResult object from a given recognition lattice. In various embodiments of the present invention, the recognition grid is a designated form of results from the recognition process. This method may, for example, designate a recognition grid as an array of bytes to be used for construction of the associated RecognitionResult object. It may also include a constructor named "RecognitionResult (ContextNode node)", which constructs a RecognitionResult object from a given context node. It may also include a constructor named "RecognitionResult (string Text, int StrokeCount)", which constructs a RecognitionResults target from a specified text value, which in turn is associated with a specified number of strokes, where the recognition process is actually If an alternative recognition value corresponding to the handwritten ink data has not been provided, it can be used for modification. In addition, this class may contain a constructor named "RecognitionResult (RecognitionResult leftRecognitionResult, RecognitionResult rightRecognitionResult)", which constructs a RecognitionResults object by merging two existing Recognition Result objects together.

The Recognition Result also provides an attribute named "StrokeCollection StrokeCollection {get;}", which provides an array of stroke indices representing a collection of strokes contained in a single ink object, and "RecognitionAlternate", which provides the best alternative to recognition results. It may contain one or more attributes, such as the one named TopAlternate {get;} ". This class also recognizes an attribute and recognition named "RecognitionConfidence RecognitionConfidence {get;}", which provides a (eg, strong, medium, or poor) reliability level of the best alternative choice of current results from the recognition analysis process. It may include an attribute named "string TopString {get;}", which returns the best result string of analysis results from the analysis process.

The Recognition Results class may also include many methods, such as the method named "public RecognitionAlternateCollection GetAlternateCollectionFromSelection (selectionStart, selectionLength, maximumAlternates)", which specifies an alternative collection from the selection in the best result string of analysis results from the recognition analysis process. Each alternative corresponds to only one ink segment. The input parameters for this method specify, for example, a value specifying the start of the text selection in which the alternative collection is returned, a value specifying the length of the text selection in which the alternative collection is returned, and a maximum number of alternatives to return. It may contain a value. This method may then return a RecognitionAlternateCollection collection of alternatives from the selection within the best result string of the recognition result.

The Recognition Results class may further include a method named "RecognitionResults Merge (RecognitionResults left, string separator, RecognitionResults right)". This method can be used to create a new RecognitionResults object from a single string that becomes a flat grid, add a single string to the beginning and end of an existing RecognitionResult object, or connect a single string between two existing RecognitionResult objects. have. This class may also include a method named "ModifyTopAlternate (RecognitionAlternate alternate)", which specifies the recognition result to be changed to known alternatives. In some embodiments of the present invention, the best result string of the results of the recognition analysis process defaults to the top alternate. However, this method can be used to specify that alternatives other than the best alternative are used in the results of the recognition analysis process. If the new best alternative is a different segment than before, the ModifyTopAlternate method will automatically update its context nodes to reflect the changes.

Note that this method calls the GetAlternatesFromSelection method, which is discussed in detail below, in order to retrieve alternatives that can be used to change the recognition result. This class can also have a method named "Stream Save ()", which keeps the associated RecognitionResults object in the form of a recognition grid. The recognition grid is an enumerated format used to represent the results from the recognition process.

The Ink Analysis API can also have Analysis Options of the type listed. This type may include one or more fields that specify how ink data will be analyzed by the analysis process, such as a field named "const AnalysisOptions Default", which enables all available options for the analysis process. . For example, this field may enable text recognition, use of tables, use of lists, use of comments, use of connectors and containers, and use of intermediate results. This type can also enable and disable detection of annotations, a field named "const AnalysisOptions EnableAnnotations", a field named "const AnalysisOptions EnableConnectorsAndContainers", which enables and disables detection of connectors and containers, and It may include a field named “const AnalysisOptions EnableIntermediateResults” that enables and disables the return of analysis results for the software application 1701 between use of other sequential analysis processes. This type may also have a field named "const AnalysisOptions EnableLists" that enables and disables the detection of the list and a field named "const AnalysisOptions EnableTables" that enables and disables the detection of the table. This enumerated type may further include a field named "const AnalysisOptions EnableTables", which enables and disables text recognition analysis processing. However, it should be noted that if additional analysis processes are available (or other versions of the same analysis process), this type may thus include additional AnalysisOptions.

The Ink Analysis API can also include an AnalysisResultsEventArgs class. This class may have a constructor named "public AnalysisResultsEventArgs ()" that contains analysis results and creates a data structure that is returned to the software application 1701 when an AnalysisResult event is raised. This class may contain an attribute named "InkAnalyzer InkAnalyzer {get;}" that identifies the InkAnalyze that performed the analysis process.

The API can also have a Line class that can be useful to several types of operating systems that recognize the use of "Line" objects that represent geometric lines. This class can contain a constructor, such as a constructor named "public Line (Point beginPoint, Point endPoint)", which creates a Line target. This class also contains an attribute named "public Point BeginPoint {get; set;}" that indicates the starting point of the line target, and an attribute named "public Point EndPoint {get; set;}" that indicates the ending point of the line target. It can contain several attributes, such as.

In addition to these classes, the Ink Analysis API may also include a Recognition Alternate class. This class may contain elements representing possible word matches for segments of the ink compared to the dictionary of the recognizer. For example, such a class might provide an ascender line for a RecognitionAlternate target that exists on a single line (the line is represented by two points), an attribute named "Line Ascender {get;}", An attribute named "public Line Baseline {get;}" that provides a baseline for an existing RecognitionAlternate object on a single line, and a descender line of an existing RecognitionAlternate object on a single line. It may contain an attribute named "Line Descender {get;}". This class may also contain an attribute named "RecognitionResult Extract {get;}", which provides RecognitionResults for the current RecognitionAlternate target. This attribute can be used, for example, to extract the RecognitionResult object for a word from the RecognitionResult object for the line containing the word.

It also provides an attribute named "Line Midline {get;}", which provides a midline for an existing RecognitionAlternate object on a single line, that is, a collection of strokes contained in the ink object (i.e. it is a RecognitionResult "StrokeCollection [] StrokesArray {which provides a collection of strokes contained in one or more ink objects representing strokes associated with a RecognitionResult and a property named" StrokeCollection Strokes {get;} " get;} ". This class also contains an attribute named "RecognitionConfidence RecognitionConfidence {get;}", which provides a recognition level (e.g., strong, medium, or poor) confidence level determined upon recognition of a RecognitionAlternate object or gesture. can do. For non-line nodes, the lowest RecognitionConfidence of the children of the relevant context nodes will be returned. It may also contain an attribute named "string RecognizedString {get;}" which specifies an alternative result string. Therefore, for any context node above the word context node, the result strings are concatenated together in this way. For example, a line node would contain a string of results that in turn contains the results of both its child nodes or word nodes. The shorting node will then contain a string of results that contains the results of both its child nodes or the line nodes.

The Recognition Alternate class also includes one or more methods, including, for example, a method named "StrokeCollection [] GetStrokeArrayFromTextRange (int selectionstart, int selectionlength)" that specifies a StrokeCollection from each ink object corresponding to a known text range. It may include. This class also includes a method named "StrokeCollection [] GetStrokesFromStrokesArrayRanges (StrokeCollection [] strokesArray)", which contains a known input collection of strokes and specifies the collection of the smallest strokes whose identifier can provide alternatives. can do. In particular, the strokes are returned by an array of ink objects that each include an array of stroke indices for that collection. It should be noted that the collection of ink strokes returned by this method may match the input collection, or may be larger if the input collection matches only some of the smallest recognition results that include all the input strokes. . This class contains a known input collection of strokes and corresponds to a method and known text range named "StrokeCollection GetStrokesFromStrokesRanges (StrokeCollection strokes)", which specifies the collection of the smallest strokes that the recognizer can provide alternatives to. The method may further include a method named "StrokeCollection GetStrokesFromTextRanges (int selectionstart, int selectionlength)" for specifying a StrokeCollection.

This class is named "void GetTextRangeFromStrokes (ref int selectionstart, ref int selectionend, StrokeCollection strokes)", which specifies the smallest range of recognized text from which the recognizer can return an alternative containing a set of known strokes. And a method named "void GetTextRangeFromStrokesArray (ref int selectionstart, ref int selectionend, StrokeCollection [] strokesarray)", which specifies the smallest range of recognized text from which the recognizer can return an alternative containing a set of known strokes. It may further include. It may also have a method named "RecognitionAlternateCollection SplitWithConstantPropertyValue (GUID propertyType)", which returns a collection of alternatives, which is the split of the alternative in which this method is called. Each alternative in the collection includes adjacent recognition segments having the same attribute value for the attribute passed in that way. For example, this method can be used to obtain alternatives for dividing the original alternative into (strong, intermediate or poor) confidence level boundaries, line boundaries or segment boundaries within the recognition result. It further includes a method named “byte [] GetPropertyValve (GUID propertyType)” that specifies the value of that alternative known attribute, such as the alternative recognizer reliability. However, not all perceptual analysis processes provide values for all attribute types. Therefore, this method provides data for the types supported by the relevant recognition analysis process.

The Ink Analysis API can also include the Recognition Alternate Collection class. Like many of the classes discussed above, such a class may include a constructor named "RecognitionAlternateCollection ()" for creating a RecognitionAlternateCollection object. This class also provides access to the attribute named "Count {get;}", the collection of alternative recognition values, which provides collections included in the collection of many objects or alternative recognition values. Namely, "IsSynchronized {get;}", which provides a value indicating whether or not to be synchronized with "thread safe", and can be used to synchronize access to a collection of alternative recognition values. It can contain many attributes, such as an attribute named "SyncRoot {get;}", which provides a target.

This class also provides a method named "virtual void CopyTo (Array array, int index)", which copies all the elements of the current collection of alternative recognition values into the specified one-dimensional array starting at the specified destination array index, and the caller ( caller) may include one or more methods, such as the method named "IEnumerable IEnumerableGetEnumerator ()", which is a standard implementation of IEnumerable that allows each configuration to be enumerated through each RecognitionAlternate in the collection of alternative recognition values. Can be. This class may also include a method named "RecognitionAlternateCollectionEnumerator GetEnumerator ()" that returns a RecognitionAlternateCollectionEnumerator that includes all objects in the collection of recognition alternative values. This method can be used, for example, to retrieve each object in a collection of recognition alternative values.

The Ink Analysis API may additionally include a Recognition Confidence enumeration and a Recognition Mode enumeration, each of which may include one or more fields related to recognition analysis processing. For example, the Recognition Confidence class might contain a field named "Intermediate" that indicates that the recognition analysis process is confident that the list is provided with alternative recognition values. Multiple fields such as a field named "Poor" indicating that the result is not certain, and a field named "Strong" indicating that the recognition analysis process is convinced that the best alternative of alternative recognition values is correct. May contain fields.

Similarly, the Recognition Mode class may include fields that specify how the recognition analysis process interprets the electronic ink data and thus how to determine the recognition result string. For example, this class may contain a field named "Coerce" specifying that recognition analysis processing dominates recognition results based on the factoroid specified for that context, and the recognition analysis processing retrieves electronic ink data as a single line. May contain a field named "Line" that specifies the handling. This class also specifies that the field named "None" specifies that the recognition analysis process does not apply any recognition modes, and that the recognition analysis process treats the electronic ink data as forming a single word or character. It may contain a field named "Segment". This class may also contain a field named "TopInkBreaksOnly", which disables multiple partitions.

The Ink Analysis API can also include a Context Link class, which defines how two context nodes can be linked together. The ContextLink object itself indicates that two context nodes, the direction of the link and the type of link are linked. This class specifies an attribute named "ContextNode SourceNode {get;}" that specifies the source context node to be linked from another context node, and a "ContextLinkType" that specifies the type of existing link relationship in the source and destination context nodes. It may include an attribute named LinkType {get;} ", and an attribute named" CustomLinkType {get;} ", which specifies that a custom link is used. This situation will occur if the application decides to use the InkAnalyzer API's linking system to represent application specific links beyond what the API can recognize.

This class also contains an attribute named "ContextNode DestinationNode {get '}", which specifies a destination context node that is linked from another context node. There can be two constructors available to this class, which creates a relationship between existing source and destination context nodes.

This class may also contain an enumeration named "ContextLinkType enum" that defines the type of relationship shared by the two context nodes. These various link types may include, for example, an "AnchorsTo" type that describes one node anchored to another node. Both nodes may use the SourceNode or DestinationNode attribute based on the parsing situation. The link types may also include type "Contains", describing that one node includes another node. With this relationship, the container node can be referred to as a SourceNode, while the container node can be referred to as a DestinationNode. The link types may further include a "PointTo" type, describing that one node points to another node. For this relationship, the pointing node may be referred to as the SourceNode, while the pointing node may be referred to as the DestinationNode. In addition, the link types may have a "PointsForm" type, describing that one node points from another node. In this relationship, node pointing away from other nodes may be referred to as a SourceNode, while nodes pointing from other nodes may be referred to as DestinationNodes.

The link types additionally include a "SpansHorizontally" type, in which one node describes the movement of another node horizontally, and a "SpansVertically", in which one node describes the movement of another node vertically. It can include a type. For these types, a node that normally covers the other node that is recorded last (strikeout, underline, margin indicator bar) may be referred to as the SourceNode, while the spanned node may be referred to as the DestinationNode. The link types may also include a "Custom" type, which describes that the on-demand link type was used. If this value is used, the "CustomLinkType" attribute for the ContextLink object may provide more details about the purpose of this link.

Thus, the Ink Analyzer API provides various functions and services for asynchronous analysis of electronic ink in a document, and then adjusts the results of the analysis process to the current state of the document as described in detail above. In addition, it should be understood that the various classes described above may apply to various operating systems and operating environments, such as the Microsoft Windows operating environment, the Microsoft COM operating environment, the Unix or Linux operating environment, or any other suitable computer operating environment. In addition, it should be understood that an application programming interface in accordance with various implementations of the invention may omit one or more of the class components described above, or may include additional components to provide a desired service or functionality.

<Adjust>

As described above, various embodiments of the present invention allow the software application 1701 to continue to operate while the unanalyzed electronic ink in its document 1703 is analyzed by the background analysis process. Because of this, the software application 1701 can change the document 1703 in a number of ways that will conflict with the results of the analysis process. For example, the software application 1701 can enter new electronic ink data 1705 into the document 1703, or delete existing electronic ink data 1705 from the document 1703. The software application 1701 can also edit the existing electronic ink data 1705 by moving the location of the existing electronic ink data 1705 or by changing the attributes of the existing electronic ink data 1705. Also, the software application 1701 can add, delete, or change non-ink document elements 1707 in ways that affect existing electronic ink data 1705. For example, the software application 1701 can delete the typed text annotated with the electronic ink data 1705.

The software application 1701 may additionally "pin" the existing electronic ink data 1705 to prohibit changes by the analysis process. For example, if a user manually specifies a layout or classification for a group of ink strokes, the software application 1701 remains as a classification where the ink strokes are within a particular layout or independent of the results generated by the parsing process. Can be specified. Similarly, a user can specify a particular recognition result for a group of ink strokes regardless of the results produced by the recognition process.

Various types of pinning can be used in accordance with other implementations of the invention. For example, the ink analysis tool 1901 may allow the software application 1701 to "hard" fix the ink. With this arrangement, no ink strokes can be added to any leaf nodes below the fixed node, and no strokes can be removed from any leaf nodes below that fixed node, and no No addition or deletion is allowed, and no re-parenting of any nodes below that fixed node is allowed. Alternatively or additionally, the ink analysis tool 1901 may allow for "hard" fixation with late strokes, which allows it to be added under specified conditions for late strokes and below the fixed node. Prohibits strokes from being removed from any leaf nodes, prohibits the addition or removal of child nodes, and prohibits re-pairing of any nodes below that fixed node. The ink analysis tool 1901 may also allow the software application 1701 to "soft" fix the ink alternatively or additionally. With this arrangement, no strokes can be removed from any leaf nodes below the fixed node, and there are rules specified to allow the addition of strokes (which allow it to be added internally for late strokes), Allow regrouping, addition and removal of child nodes. It should be noted that pinning can be removed, and once pinning is removed, previously pinned nodes and their children can be considered "dirty" or require reanalysis.

In addition, the ink process according to various embodiments of the present invention may utilize multiple analysis processes, as discussed in detail above. Thus, while the second next analysis process is being executed, the results of the previous analysis process can change the electronic ink data 1705. Thus, since the results of the analysis process must be adjusted to the current state of the document 1703, only those results valid for the current state of the document 1703 are applied to the analysis context object 1709. That is, the current analysis context object 1709 (and in some cases the results of the analysis process) omits discrepancies or “collisions” between the results of the analysis process and the current state of the document 1703. To change it.

Note that, in order for the results of the adjustment process to be valid, the state of the document 1703 should not be changed during the adjustment process. Therefore, the adjustment process can be performed using the primary thread on which the software application 1701 is executed, and executing the adjustment process can temporarily stop the operation of the software application 1701. Alternatively, other techniques, such as data structure locking, can be used to ensure that the state of the document 1703 does not change during the reconciliation process. Therefore, in order to prevent the user from becoming unsatisfied with the performance of the software application 1701, it is preferable to perform the adjustment process as soon as possible. Another consideration of the tuning process is the effect on the performance of the analysis process running on a separate background analysis thread. If inconsistencies between the current analysis context object 1709 (ie, the analysis context object 1709 reflecting the current state of the document 1703) and the analysis results are defined too broadly, a large amount of electronic ink will be unnecessarily reanalyzed. Of course, other ways of protecting the integrity of the document during the adjustment can be used without departing from the present invention.

As various embodiments of the present invention, the analysis process and the adjustment process may cling to one or more of the following conventions to improve the efficiency and convenience of the adjustment process. First, the analysis process can reuse as many nodes as possible for document elements in the document independent analysis context object 1903. That is, conflicts should not be avoided by creating new unrelated nodes for document elements. In addition, the adjustment process must relate to "pinning" specified by the software application 1701. While the analysis process will typically follow the designations of fixation by the software application 1701, the software application 1701 can fix the electronic ink data 1705 while the analysis process is executed. In addition, the coordination process must ensure that no empty nodes remain in the current analysis context object 1709 after the coordination process is completed. However, it should be understood that one or more of these conventions may be omitted and may not be followed in accordance with alternative implementations of the invention. For example, some implementations of the invention may allow the subject 1709 to include empty nodes.

In addition to these conventions, the coordination process should typically follow the interface rules specified by the analysis context object 1709. For example, in some implementations of the invention, the analysis context 1709 may not allow a node for a document element to be deleted unless it has any child nodes.

As mentioned above, conflicts arise when the analysis process changes for the document independent analysis context object 1903 as opposed to some form of change to the analysis context object 1709 made after the analysis process has begun. Collisions can be divided into two types: mandatory collisions and unconditional collisions.

Forced conflicts arise when it is impossible to apply a change made to the document independent analysis context object 1901 by the analysis process for the analysis context object 1709 to the current state of the document 1703. For example, the software application 1701 "pins" or fixes a node to an analysis context object 1709, and the analysis process changes the corresponding node within the document independent analysis context object 1903. In this case, a forced collision occurs. A mandatory conflict also occurs when the analysis process makes any type of change to a node in the document independent analysis context object 1901 but the software application 1701 deletes the corresponding node from the analysis context object 1709, And when the software application 1701 adds strokes or child nodes to the node in the analysis context object 1709 when the analysis process deletes the corresponding node in the document independent analysis context object 1901. In addition, if the analysis process deletes a node in the software application 1701 analysis context object 1709 when it reorders or creates a link to the corresponding node in the document independent analysis context object 1903, it is forced. A crash will occur.

When the software application 1701 changes the value in the analysis context object 1709 related to the value changed in the document independent analysis context object 1901 by the analysis process, an unconditional conflict occurs, but fixed constraints, element reuse constraints. And the inherent constraints of the interface to the analysis context object 1709 may allow application of changes made by the analysis process to the analysis context object 1709. Any analysis can also be applied. Unconditional collision can be applied or avoided as a change to the analysis context object 1709. The coordination process can also simply ignore some types of unconditional collisions.

One graphical example of an unconditional collision occurs when the first node of both analysis context object 1709 and document independent analysis context object 1901 has child nodes A and B for ink strokes A and B. The software application 1701 can then add a third child C node for ink stroke C within analysis context object 1709, but the analysis process removes the ink stroke D from document independent analysis context object 1903. 3 Add child node D. In various embodiments of the present invention, the adjustment process may add the child node D to the parent node in the analysis context object 1709 based on the analysis results. However, even if such a change to the parent node in the analysis context object 1709 is not prohibited, the characteristics of the parent node are changed in a manner that may not be desired by the software application 1701. For example, the ink associated with its parent node may then be analyzed and considered and may not be reanalyzed to account for the effect of ink C on the group of ink strokes. Also, recognition results for the parent node may now refer to erroneous child nodes or strokes, and may never be modified again.

Thus, when various embodiments of the present invention update the analysis context object 1709 based on the results of the analysis process, no changes to unconditional collisions will be applied. Although this criterion may require additional processing to identify and block changes based on unconditional conflicts, this criterion is relatively easy to implement and simple to maintain. Of course, other examples of the present invention may implement changes corresponding to unconditional collisions according to other criteria. In particular, examples of such an alternative may implement changes from unconditional conflicts that do not create permanent logical inconsistencies in the analysis context object 1709.

Note that collisions can have a transitive effect in that one collision can create another collision. For example, the analysis process may generate node L for the line and then generate node W for the word as a child node of node L. If the creation of the node L was not applied to the analysis context object 1709 due to some kind of collision, the creation of the node W would be a forced collision.

Various embodiments of the present invention may use a log-based approach to adjust the results of the analysis process to the current state of the document 1703. In this log-based approach, the document independent analysis context object 1903 includes a log of changes to the document independent analysis context object 1903 made by the analysis process. The log may be in the form of a list of change records, for example. Each change is then a type of change made (e.g., by identifying a method that was invoked to change the document independent analysis context object 1903), and the change is made (e.g., the document independent analysis context from which the method was invoked). Document elements), and any information required to recreate arguments for the method call. Advantageously, because the document independent analysis context object 1903 is implemented by the ink analysis tool 1901 and analysis processes (although the log may alternatively be exposed to the application 1701 if desired), The change log may not be visible to the software application 1701.

In order to perform the adjustment process using the change log approach, the ink analysis tool 1901 examines each change record in chronological order of the changes. That is, the ink analysis tool 1901 can implement the processing shown in FIG. First, at step 2701, the ink analysis tool 1901 attempts to access corresponding nodes in the current analysis context object 1709 that are required to apply the change. It should be noted that this search step may fail if the software application 1701 fails if it deletes one or more required nodes from the forcing conflicting analysis context object 1709.

Next, at step 2703, the ink analysis tool 1901 determines whether the change produces a forced collision or an unconditional collision. The procedure for making this determination will be described in more detail below. In step 2705, the ink analysis tool 1901 makes a change or prohibits the change if it creates a forced collision or an unconditional collision prohibited by the criteria for the adjustment process. For example, if the change creates a forced conflict or an unconditional conflict that is prohibited by the criteria for the adjustment process, the ink analysis tool 1901 may display an analysis context that indicates the corresponding area of the document 1703 in which the change was made. Changes can be blocked for nodes within the target 1709. On the other hand, once the change is applied, the ink analysis tool 1901 can invoke the appropriate method to make the desired change to the necessary nodes in the analysis context object 1709.

If the analysis process adds a new element node to analysis context object 1709 but then fails to move the stroke nodes to the new node, the analysis process also assumes that the stroke nodes have been successfully moved to the new nodes. You will not want to delete the element node. Therefore, there will be an element node in the document independent analysis context object 1903 corresponding to an empty node in the current analysis context object 1709. Thus, once all changes to the analysis context object 1709 have been processed, at step 2707, the ink analysis tool 1901 may analyze the analysis context object 1709 corresponding to the nodes in the document independent analysis context object 1903. Review the document independent analysis context object to delete any "empty" nodes within. This empty node deletion step is optional and can be omitted without departing from the present invention.

It should be noted that the document independent analysis context object 1903 should not include any empty nodes, and that the adjustment process should prevent empty nodes from being created in the document independent analysis context object 1903. It should also be noted that this step will not attempt to delete the root node of the analysis context target 1709 even if it is empty as an exception. Finally, at step 2709, the ink analysis tool 1901 identifies any conflict between the document independent analysis context target 1901 and the analysis results of the current analysis context target 1709 for the software application 1701. As such, the software application 1701 may include areas of the document 1703 that are affected by conflicts in the next analysis process.

Returning to the processing of conflicts in step 2703 now, each node in analysis context object 1709 once all nodes in analysis context object 1709 corresponding to changes in document independent analysis context object 1903 are accessed. All other possible coercive conflicts for C may be statically deleted (or alternatively, the absence of coercive collisions may be deleted for each node). In particular, mandatory conflicts can be deleted based on rules specified by the interface to the analysis context object 1709.

However, these changes are not enforced by the interface of the analysis context object 1709, but instead, if both affect the same node within the analysis context object 1709, the software application (i.e. Based on unconditional choices for friendly changes made by 1701, unconditional conflicts cannot be identified without additional state information about that document 1703. Instead, these conflicts compare the current analysis context object 1709 with the original version of the document independent analysis context object 1901 to determine that nodes within the analysis context object 1709 have been modified by the software application 1701. Can be deleted.

For example, the coordination criteria may define that the addition or deletion of child stroke nodes from the parent node by the software application 1701 is an unconditional collision for that parent node. To determine if such a change occurs, the ink analysis tool 1901 reviews the child stroke nodes that depend from the node within the current analysis context object 1709 and corresponds to the original version of the document independent analysis context object 1903. You can compare their number and Guid (or other unique identifier) to the child stroke nodes that depend from that node. If the child stroke nodes match these embodiments (and any other desired embodiments), the ink analysis tool 1901 can make changes made by the analysis processes to the parent node within the analysis context object 1709. Applicable

Subsequent to this example, the next change in the log entry may be made with another stroke node that is moved to rely on the parent node in the document independent analysis context object 1903. In the absence of any mandatory collision from these changes, the ink analysis tool 1901 is allowed to make a change corresponding to the analysis context object 1709. However, if the ink analysis tool 1901 checks the first version of the document independent analysis context target 1901 to determine if an unconditional conflict exists, this means that the parent node within the current analysis context target 1709 is analyzed by the document independent. It will be concluded that there are more than one dependency stroke node than the parent node in the original version of the context target 1903, which would therefore incorrectly determine that a mandatory conflict exists (i.e., based on the changes made) Will decide).

Thus, the log-based reconciliation process should exclude previous changes made by the reconciliation from the comparison between the current version of the document independent analysis context object 1901 and the original version of the analysis context object 1709. In various embodiments of the present invention, such exclusion may include any of their corresponding nodes within the original version of the document independent analysis context object 1903 before examining any list entries and maintaining a list of conflicting nodes. This can be done by comparing all nodes in the current analysis context object 1709.

Alternatively or additionally, various embodiments of the present invention may be used whenever an element corresponding to a cache of elements in the current analysis context object 1709 corresponds to elements in the document independent analysis context object. The results of the analysis can be maintained. There is no temporary delivery, but the cache is populated at the beginning when the corresponding element is determined to have unconditional collisions. Optionally, the cache can also keep track of which elements of the results had unconditional collisions. This approach has better performance than the discretionary-collision-finding-pass approach when the change log is short, but if the document independent analysis context target 1903 with analysis results includes many nodes. You may have worse performance. It is also not more complicated than any other approach.

Note that the parent context nodes of the elements in the cache should also be cached. In particular, a change to a child context node will be considered a change to the attributes of its parent context node. Therefore, the parent context node of the context node stored in the cache also needs to be evaluated for changes made by the software application 1701 at this moment. Caching of the parent nodes for each context node in the cache is done until the root node is delivered or until the changes are detected (eg, from the parent node to the grandparent to great-grandparents). (grandparent) until the node needs to be repeated. However, in many cases, this iteration only repeats a few nodes, since the tree will typically not be very deep.

Also as various nodes of the present invention, the original version of the document independent analysis context object 1903 is only used to detect unconditional conflicts. If detection of unconditional conflicts is not essential as examples of the present invention, both the document independent analysis context object 1903 and the corresponding element cache may be removed. This "log-based" approach requires less searching to determine what changes have been made by the analysis process and because the operations for the document independent analysis context 1903 are handled separately, the " It may be easier to maintain than a "comparison-based" approach, and therefore adding a new type of operation is not easy to affect other operations.

Instead of a "log-based" technique for adjusting the analysis results to the current state of document 1703, various embodiments of the present invention may use a "comparison-based" approach for adjustment. The main distinguishing feature of the comparison-based approach is that no log of changes made to the document independent analysis context object 1903 is maintained, so this technique uses document independent analysis context objects containing the analysis results as document independent. It does not collect information about what the analysis process did by comparing it with the original version of the analysis context object 1903. Therefore, with this approach, the first version of document independent analysis context object 1903 is always required, regardless of any knowledge of unconditional conflicts.

Using this approach, the ink analysis tool 1901 first builds a stroke map. That is, for every ink leaf node in the original version of the document independent analysis context object 1903, if there is a corresponding node in the current analysis context object 1709, the leaf node is a hash table (or other appropriate data). Structure). Therefore, the hash table maps the stroke GUIDs (or other single node identifiers) in the original version of the document independent analysis context object 1903 to leaf node references in the current analysis context object 1709.

Next, the ink analysis tool 1901 identifies all nodes that the software application 1701 changes in the analysis context object 1709. This is done using techniques for determining the unconditional collisions described in detail above. For each node in the original version of the document independent analysis context object 1903, there is a corresponding node within the current analysis context object 1709 and it is in some important way the node in the original version of the document independent analysis context object 1903. If different, the ink analysis tool 1901 determines that this node is modified by the application and is a potential conflict. References to such nodes should be added to the first list corresponding to analysis result nodes that should not be added to the analysis context object 1709 (hereafter referred to as resultsNodesNotToAdd list) and to the second list of nodes generating unconditional conflicts. Stored.

For each node in the original version of the document independent analysis context object 1903, the ink analysis tool 1901 is in both the document independent analysis context object 1903 and the current analysis context object 1709 having the analysis results. Find the corresponding nodes. For each node in analysis context object 1709, the ink analysis tool 1901 determines whether the software application 1701 has deleted the node. Such nodes exist in the original version of document independent analysis context object 1903, but there is no corresponding node in analysis context object 1709. The corresponding node in the document independent analysis context object 1903 with the analysis results is added to the resultsNodesNotToAdd list if it exists.

Next, the ink analysis tool 1901 determines whether the node has been deleted from the document independent analysis context object 1903. This is detected when there is a corresponding node in analysis context object 1709 and it has not changed, but there is no corresponding node in document independent analysis context object 1903 with its analysis results. Nodes deleted by the analysis process are stored in the list of deleted nodes. Such a node is not currently added to the analysis context object 1709, but is not removed from its resultsNodesNotToAdd list because there is no corresponding node in the document independent analysis context object 1901 with analysis results.

If the software application 1701 does not delete or change the node, and the analysis process does not delete the node, the ink analysis tool 1901 can determine whether the analysis process has changed the node. The same techniques as described above for use in detecting changes made by software application 1701 can also be used to detect changes made by the analysis process. Nodes changed by the analysis process are added to the resultsNodesNotToAdd list, and the changes are propagated by invoking the nodes in the analysis context object 1709 in an appropriate manner. If neither the software application 1701 nor the analysis processing node has changed or deleted, then nothing happens to that node, and that node exists in the document independent analysis context target 1901 having the analysis results and that analysis context target 1709. The ink analysis tool 1901 adds the node to the resultsNodesNotToAdd list.

Next, for each node in the document independent analysis context object 1901 with the results of the analysis, if the node is not in the resultsNodesNotToAdd list, the ink analysis tool 1901 will correspond to the corresponding node in the analysis context object 1709. It is possible to propagate any changes to that node by the analysis process by generating. The traversal of the nodes in the analysis results can be performed using top-down pre-order search, so that parent nodes are guaranteed to be created before child nodes.

When creating a new leaf node in the analysis context object 1709, any ink strokes associated with the node included in the analysis results may also be moved from other nodes in the analysis context object 1709 to the new nodes. The stroke map discussed above can then be used to identify source elements for moving the strokes. The ink strokes are not moved if there is no source element (creating a mandatory collision), or if the source element is included in the aforementioned unconditional collision list. To store unnecessary calls to the analysis context object 1709, this ink analysis tool 1901 calls the method to create a child context node if there is no at least one non-colliding stroke source node. Can be avoided. The same stroke source check is also performed at non-ink leaf nodes, so that no node for a line is created if the component words of at least one line will not contain, for example, at least one successfully moved stroke. .

Subsequently, other information is also applied to the newly created node, such as the recognition grid. The recognition grid applies even if any strokes that result in an applied (ie non-blocked) unconditional collision resulting in temporary logical inconsistency cannot be moved. However, because the software application 1701 is notified to reanalyze the blocked area after the failed stroke movements are reflected in the designation of the portion of the analysis context object 1709 that corresponds to the area of the document where the changes are blocked, the inconsistency is inconsistent. It is temporary.

For each node to be deleted from the analysis context object 1709, the ink analysis tool 1901 list adds all other edits and additions to the analysis context 1709 to see if the node can actually be deleted. Check that they are performed. Nodes that must be deleted and all child nodes and strokes are actually empty are deleted from the analysis context object 1709. Once such a node is deleted, all ancestor nodes (up to the root node but not the root node) are deleted continuously until the ancestors encounter other remaining children.

Finally, the ink analysis tool 1901 is capable of detecting and propagating changes to the analysis context object 1709 by the analysis process, so that the document independent analysis context object 1903 and analysis context object 1709 having the analysis results. Compare nodes that are ordering between However, it should be noted that this comparison ignores the original version of the document independent analysis context object 1901 and so has no way of detecting changes to the ordering made by the software application 1701. Next, for each container node in the analysis results having a corresponding node in the analysis context object 1709, the ordering of the children of those container nodes is compared. If both container nodes have exactly the same set of children, the ordering of the analysis results is a list of containers in the analysis context object 1709 where the node does not have the same GUID (or other identifier type) as the child node of the analysis results. It is propagated to the analysis context object 1709 by looping through the children of the container in its analysis results until found at the same location within. If such a mismatch is found, then the corresponding node must also be found in the child list within that analysis context object 1709. The ink analysis tool 1901 then retrieves the remainder of the child list in the analysis context target 1709 for that node and moves the child node to modify the command for the node in the analysis context 1709. Call The ink analysis tool 1901 then loops through the nodes in the analysis results.

Of course, the assumptions of the same lists remain true. The analysis list or software application 1701 may delete or insert nodes in any list. A "child list mapper" object is used to simulate the assumption of keeping true by presenting "pruned" lists that only contain elements that are kept in common. The previously described procedure is then executed on the mapped child lists, and the movements are moved to actual indices and calls by the list mapper object.

Therefore, the adjustment techniques in accordance with various embodiments of the present invention described above allow the results of the analysis process to be applied to current document elements in the document even if the contents of the document have changed after the analysis process has begun. Therefore, these adjustment techniques allow the electronic ink in the document to be analyzed asynchronously from the operation of the software application hosting the document. Such coordination techniques may also be used by a variety of other software applications, including software applications multithreaded with existing proprietary locking or other concurrency strategies.

<Event Drive System>

As discussed above, various embodiments of the present invention create a "snapshot" of the document state by copying a document independent analysis context object, while the software application hosting the document continues to operate, Independent Analysis of Documents Analyzes context objects asynchronously. Alternatively, various embodiments of the present invention may precede the use of a document independent analysis context object for asynchronous ink analysis. Instead, these examples of the invention may use the identified component to store all the ink and semantic data for the document. In particular, these examples of the invention include documents: ink events, such as adding, deleting, or changing strokes; And two types of changes that can be made to structural events such as grouping strokes into words, adding semantic nodes, or associating text recognition results to strokes. Every event contains all the data necessary to completely describe the event to an external listener. For example, the "Add Stroke" event will contain all the stroke data. As such "rich" events, the listener can maintain an accurate copy of the ink object by applying the events to the command they received.

Figure 28 illustrates one example of how this arrangement may be used in accordance with various embodiments of the present invention. As shown in this figure, the application 2801 uses the ink analysis tool 2803. The ink analysis tool 2803 maintains ink data and document structure 2807 (such as, for example, a tree structure) for the application 2801. The application also utilizes an event queue 2809, parser processing 2811 and recognition processing 2817. The parser processing 2811 holds a clone of the ink data 2805 and a clone 2815 of the document structure 2807. Similarly, the recognition process 2817 holds a clone 2819 of the ink data 2805 and a clone 2821 of the document structure 2807.

If the application generates ink data, it provides ink data to the ink analysis tool 2803 via method 2823. In response, the ink analysis tool 2803 generates an event corresponding to a change in the ink data, adds a tag to all events, and defines its desired analysis process as components that follow the events with specific tags. . For example, all users may be marked with the tag "UserChange" and an event 2825 with this tag is sent to its event queue 2809 simultaneously in response to the change. At some point in the future, the parser process 2811 and recognition process 2817 will retrieve the tagged event from the event queue 2809.

The first analysis process (eg, parsing process 2811) will then follow all the events issued by that ink analysis process 2803 and respond to them with the tag UserChange. The first analysis process then applies the changes specified in the event to an internal copy of the ink data 2813 and document structure 2815, analyzes the data described in the event, and then by those changes with the tag "Parser1Change". Generated events 2827 will be generated and tagged. These events 2827 will return to the event queue 2809 and be relayed by the ink analysis tool 2803. The second analysis process (e.g., handwriting recognition process 2817) will then follow the events tagged with the Parser1Change tag. In response, this event is included in an internal copy of the ink data 2819 and the document structure 2821. Apply the changes specified by and respond to the data described in those events The second analyzer will then generate and tag event 2829 with analysis results with the tag "HandwritingRecognitionChange". When tool 2803 receives events from parser processing 2811 and recognition processing 2817, it sends events 2831 to application 2801 to indicate a change in document structure 2807.

These embodiments of the present invention may also support each analysis process by following events with one or more events, and tag those events from changes with other tags based on internal processing. Thus, these alternative implementations of the present invention may also consider the analysis of electronic ink in a document asynchronously from the operation of a software application hosting the document.

Conclusion

Although the present invention has been described with respect to specific examples which now include the preferred modes of carrying out the invention, those skilled in the art will appreciate that systems and techniques described above that fall within the spirit and scope of the invention as set forth in the appended claims. It will be understood that there are various changes and exchanges.

Claims (18)

As a method of analyzing electronic ink, Receiving document data for a document containing electronic ink content from a software application running on a first processing thread; Employing the first processing thread to provide the document data to an electronic ink analysis process for analysis on a second processing thread; Returning control of the first processing thread to the software application; Receiving a result of the analysis process; And Adjusting the results of the analysis process with current document data for the document How to include. The method of claim 1, Receiving the adjusted analysis result from the software application running on the first processing thread; Employing the first processing thread to provide the adjusted analysis results to a second electronic ink analysis process for analysis on a third processing thread; Returning control of the first processing thread to the software application; Receiving a result of the second analysis process; And Adjusting the result of the second analysis process with current document data for the document How to include more. The method of claim 2, Wherein the first analysis process is an electronic ink layout and classification analysis process and the second analysis process is a recognition process. The method of claim 2, The third processing thread is the same as the second processing thread. The method of claim 2, Receiving the adjusted second analysis result from the software application running on the first processing thread; Employing the first processing thread to provide the adjusted second analysis result to a third electronic ink analysis process for analysis on a fourth processing thread; Returning control of the first processing thread to the software application; Receiving a result of the third analysis process; And Adjusting the result of the third analysis process with current document data for the document How to include more. The method of claim 2, And said second analysis process is a recognition process having a first stage for recognizing electronic ink data designated to be a first language and a second stage for recognizing electronic ink data designated to be a second language. The method of claim 1, The document data includes ink content and non-ink content of the document. The method of claim 7, wherein At least a portion of the electronic ink content annotates the non-ink content. The method of claim 1, Generating a data structure for the document data received from the software application; And Providing the data structure to the first analysis process How to include more. The method of claim 9, Providing the data structure to the software application for use in maintaining the current state of the document. A computer readable medium comprising computer executable instructions for performing a method according to claim 1. As a software operating environment for the analysis of electronic ink, A software application for maintaining a document including document data including electronic ink data; An ink analysis process for analysis of electronic ink; And An ink analysis tool, comprising: receiving the document data comprising electronic ink data from the software application, providing the document data to the electronic ink analysis process for analysis, and generating the result by the analysis process to the software application Ink analysis tool Software operating environment comprising a. The method of claim 12, The software application runs on a first processing thread, And the analysis process operates on a second thread different from the first thread such that the software application continues to operate while the analysis process analyzes the electronic ink in the document data. The method of claim 13, And the ink analysis tool adjusts the results produced by the analysis process with current document data for the document. The method of claim 1, A second analysis process for analyzing the electronic ink, The ink analysis tool provides the results generated by the first analysis process to the second analysis process for analysis, A software operating environment for returning a result produced by the second analysis process to the software application. The method of claim 15, The ink analysis tool adjusts the result generated by the first analysis process with current document data for the document, A software operating environment for providing the adjusted results generated by the first analysis process to the second analysis process for analysis. The method of claim 16, And the ink analysis tool adjusts the results generated by the second analysis process with current document data for the document. The method of claim 15, And the ink analysis tool adjusts the results generated by the second analysis process with current document data for the document.

Images