JP2009540404A - Contextual dynamic ads based on captured rendering text - Google Patents

Abstract

A system for processing text captured from a rendered document is described. The system receives a sequence of one or more words that are optically or acoustically captured by a user from a rendered document. The system identifies an electronic version of the rendered document and associates actions. The system then performs the relevant action on the user.
[Selection] Figure 1

Description

(Cross-reference to related applications)
This application is incorporated by reference herein in its entirety by “TRIGGERING ACTIONS IN RESPONSE TO OPTICALLY OR ACOUSTICLY CAPTUREING KEYWORDS FROM A RENDERED DOCUMENT that incorporates an optically or acoustically triggered response from a rendered document. Is a continuation-in-part application of US patent application Ser. No. 11 / 097,103, filed Apr. 1, 2005.
This application was filed on June 6, 2006 under the name “CONTEXTUAL DYNAMIC ADVERTISING BASED UPON CAPTURED RENDERED TEXT”, which is incorporated herein by reference in its entirety. Claims priority to US Provisional Patent Application No. 60 / 811,623.
This application is related to the following US patent applications, each of which is incorporated herein by reference in its entirety:
US patent application Ser. No. 11 / 004,637 filed Dec. 3, 2004;
US patent application filed April 1, 2005 under the name "METHODS AND SYSTEMS FOR INITITING APPLICATION PROCESSES BY DATA CAPTURE FROM RENDERED DOCUMENTS". No. 11 / 097,961,
"DETERMINING ACTIONS INVOLVING CAPTURED INFORMATION AND ELECTRONIC CONTENT ASSOCIATED WITH RENDERED DOCUMENTS" 11 / 097,093,
US patent application Ser. No. 11 / 098,038 filed Apr. 1, 2005, under the name “CONTENT ACCESS WITH HANDHELD DOCUMENT DATA CAPTURE DEVICES”.
US patent application Ser. No. 11 / 098,014, filed Apr. 1, 2005, under the name “SEARCH ENGINES AND SYSTEMS WITH HANDHELD DOCUMENT DATA CAPTURE DEVICES (search engine and system with handheld document data capture device)”;
US application filed April 1, 2005 under the name "SEARCHING AND ACCESSING DOCUMENTS ON PRIVATE NETWORKS FOR USE WITH CAPTURES FROM RENDERED DOCUMENTS". Patent application No. 11 / 098,043,
US patent application Ser. No. 11 / 097,981, filed Apr. 1, 2005 under the name “INFORMATION GATHERING SYSTEM AND METHOD”.
US patent application Ser. No. 11 / 097,089, filed Apr. 1, 2005 under the name “DOCUMENT ENHANCEMENT SYSTEM AND METHOD”.
US patent application Ser. No. 11 / 097,835, filed Apr. 1, 2005, under the name “PUBLISHING TECHNIQUES FOR ADDING VALUE TO A RENDERED DOCUMENT”.
US patent application Ser. No. 11 / 098,016, filed Apr. 1, 2005, entitled “ARCHIVE OF TEXT CAPTURES FROM RENDERED DOCUMENTS”;
"ADDING INFORMATION OR FUNCTIONALITY TO A RENDERED DOCUMENT VIA ASSOCIATION WITH AN ELECTRONIC COUNTERPART (added to the United States on the date of the 5th month of the application in the United States on the date of the 5th year)" Patent application No. 11 / 097,828,
US Patent Application No. 11 filed on April 1, 2005 under the name "AGGREGATE ANALYSIS OF TEXT CAPTURES PERFORMED BY MULTIPLE USERS FROM RENDERED DOCUMENTS" 097,833,
US patent application Ser. No. 11 / 097,836, filed Apr. 1, 2005, entitled “ESTABISHING AN INTERACTIVE ENVIRONMENT FOR RENDERED DOCUMENTS”;
US patent application Ser. No. 11 / 098,042, filed Apr. 1, 2005, entitled “DATA CAPTURE FROM RENDERED DOCUMENTS USING HANDHELD DEVICE”.
US patent application Ser. No. 11 / 096,704, filed Apr. 1, 2005, under the name “CAPTURING TEXT FROM RENDERED DOCUMENTS USING SUPPLEMENTAL INFORMATION”.
US patent application Ser. No. 11 / 110,353 filed Apr. 19, 2005, entitled “PROCESSING TECHNIQUES FOR VISUAL CAPTURE DATA FROM A RENDERED DOCUMENT”. issue,
US patent application Ser. No. 11 / 131,945 filed on May 17, 2005 under the name “PROCESSING TECHNIQUES FOR TEXT CAPTURE FROM A RENDERED DOCUMENT”.
US patent application Ser. No. 11 / 185,908 filed Jul. 19, 2005 under the name “AUTOMATIC MODIFICATION OF WEB PAGES”.
US patent application Ser. No. 11 / 208,408, filed Aug. 18, 2005, entitled “SCANNER HAVING CONNECTED AND UNCONNECTED OPERATIONAL BEHAVIRS”.
"Fragment of LOCATING ELECTRONIC INSTANCES OF DOCUMENTS BASED ON RENDERED INSTANCES, DOCUMENT FRAGMENT DIGEST GENERATION, DIGEST NAME NAME US patent application Ser. No. 11 / 208,457, filed Aug. 18, 2005,
"METHODS, SYSTEMS AND COMPUTER PROGRAM PRODUCTORS FOR DATA GATHERING IN A DIGITAL AND HARD COPY DOCUMENT ENVIRONMENT" US patent application Ser. No. 11 / 208,458 filed on
US patent application Ser. No. 11 / 208,461, filed Aug. 18, 2005, under the name “APPLYING SCANNED INFORMATION TO IDENTIFY CONTENT”.
US patent application Ser. No. 11 / 209,333, filed Aug. 23, 2005 under the name “A PORTABLE SCANNING DEVICE”,
US patent application Ser. No. 11 / 210,260, filed Aug. 23, 2005 under the name “A METHOD AND SYSTEM FOR CHARACTOR RECOGNITION”;
US patent application Ser. No. 11 / 236,440, filed Sep. 27, 2005, under the name “SECURE DATA GATHERING FROM RENDEDED DOCUMENTS”.
"HANDHELD DEVICE FOR CAPTUREING TEXT FROM BOTH A DOCUMENT PRINTED ON PAPER AND A DOCUMENT DISPLAYED ON A DYNAMIC DISPLAY DEVICE ON A DYNAMIC DISPLAY DEVICE ON A DYNAMIC DISPLAY US patent application Ser. No. 11 / 236,330, filed Sep. 27, 2005,
US patent application Ser. No. 11 / 365,983, filed Feb. 28, 2006 under the name “ASSOCIATION OF A PORTABLE SCANNER WITH INPUT / OUTPUT AND STORE DEVICES”. issue,
US patent application Ser. No. 11 / 432,731, filed May 11, 2006 under the name “PORTABLE SCANNING AND MEMORY DEVICE”;
International Patent Application No. PCT / US05 / 11533 filed on April 1, 2005 under the name “A SYSTEM AND METHOD THE FOR CHARACTOR RECOGNITION”,
International Patent Application No. PCT / US05 / 13586, filed April 6, 2005 under the name “SCANNING APPARATUS AND RELATED TECHNIQUES”,
International Patent Application No. PCT / US05 / 12510 filed on April 12, 2005 under the name "ADDING VALUE TO A RENDERED DOCUMENT".

This application is further related to the following US provisional patent applications, all of which are incorporated herein by reference:
US application 60 / 559,226 filed April 1, 2004, application 60 / 558,893 filed April 1, 2004, application filed April 1, 2004 No. 60 / 558,968, Application No. 60 / 558,867 filed on April 1, 2004, No. 60 / 559,278 filed Apr. 1, 2004, April 2004 Application 60 / 559,279 filed on 1st, Application 60 / 559,265 filed on 1st April 2004, Application 60/559, filed 1st April 2004, No. 277, Application No. 60 / 558,969 filed on Apr. 1, 2004, No. 60 / 558,892 filed Apr. 1, 2004, filed Apr. 1, 2004. No. 60 / 558,760, 2 Application No. 60 / 558,717 filed on April 1, 2004, Application No. 60 / 558,499 filed on April 1, 2004, Application No. filed on April 1, 2004 No. 60 / 558,370, Application No. 60 / 558,789 filed on Apr. 1, 2004, No. 60 / 558,791 filed Apr. 1, 2004, Apr. 1, 2004 Application 60 / 558,527, filed on April 2, 2004, Application 60 / 559,125, filed April 2, 2004, Application 60 / 558,909, filed April 2, 2004 No. 60 / 559,033 filed on Apr. 2, 2004, No. 60 / 559,127 filed Apr. 2, 2004, filed Apr. 2, 2004 Application No. 60 / 559,087, 2004 Application No. 60 / 559,131 filed on Apr. 2, Application No. 60 / 559,766 filed Apr. 6, 2004, Application No. 60 / filed Apr. 12, 2004 No. 561,768, Application No. 60 / 563,520 filed on April 19, 2004, No. 60 / 563,485, filed Apr. 19, 2004, April 23, 2004. Application No. 60 / 564,688, Application No. 60 / 564,846 filed on Apr. 23, 2004, Application No. 60 / 566,667 filed Apr. 30, 2004, Application No. 60 / 571,381 filed on May 14, 2004, Application No. 60 / 571,560 filed on May 14, 2004, Application No. filed on May 17, 2004 60 / 571,715 Application No. 60 / 589,203 filed on July 19, 2004, Application No. 60 / 589,201 filed on July 19, 2004, Application filed on July 19, 2004 No. 60 / 589,202, Application No. 60 / 598,821 filed on August 2, 2004, No. 60 / 602,956, filed Aug. 18, 2004, August 2004 Application 60 / 602,925 filed on 18th, Application 60 / 602,947 filed 18th August 2004, Application 60/602 filed 18th August 2004 No. 897, Application No. 60 / 602,896 filed on August 18, 2004, Application No. 60 / 602,930 filed Aug. 18, 2004, filed August 18, 2004 Application No. 60/6 No. 2,898, Application No. 60 / 603,466, filed Aug. 19, 2004, Application No. 60 / 603,082, filed Aug. 19, 2004, Aug. 19, 2004. Application No. 60 / 603,081, Application No. 60 / 603,498, filed Aug. 20, 2004, Application No. 60 / 603,358, filed Aug. 20, 2004, Application No. 60 / 604,103 filed on Aug. 23, 2004, Application No. 60 / 604,098 filed on Aug. 23, 2004, Application No. filed on Aug. 23, 2004 No. 60 / 604,100, Application No. 60 / 604,102 filed Aug. 23, 2004, No. 60 / 605,229 filed Aug. 27, 2004, Aug. 27, 2004. Filed on the day Application No. 60 / 605,105, Application No. 60 / 613,243 filed on Sep. 27, 2004, Application No. 60 / 613,628 filed Sep. 27, 2004, No. 2004 Application No. 60 / 613,632, filed on May 27, Application No. 60 / 613,589, filed Sep. 27, 2004, Application No. 60/613, filed Sep. 27, 2004 No. 242, Application No. 60 / 613,602 filed on Sep. 27, 2004, No. 60 / 613,340 filed Sep. 27, 2004, filed Sep. 27, 2004 Application No. 60 / 613,634, Application No. 60 / 613,461 filed on Sep. 27, 2004, Application No. 60 / 613,455 filed Sep. 27, 2004, 2004 September 2 Application 60 / 613,460 filed on 7th, Application 60 / 613,400 filed on 27th September 2004, Application 60/613 filed on 27th September 2004 No. 456, Application No. 60 / 613,341 filed on Sep. 27, 2004, No. 60 / 613,361 filed Sep. 27, 2004, filed Sep. 27, 2004. Application No. 60 / 613,454, Application No. 60 / 613,339 filed on Sep. 27, 2004, Application No. 60 / 613,633 filed Sep. 27, 2004, 2004 Application 60 / 615,378, filed October 1, Application 60 / 615,112, filed October 1, 2004, Application 60/615, filed October 1, 2004 615,538, Application No. 60 / 617,122 filed on Oct. 7, 2004, Application No. 60 / 622,906 filed on Oct. 28, 2004, Application No. filed on Dec. 6, 2004 No. 60 / 633,452, Application No. 60 / 633,678, filed Dec. 6, 2004, No. 60 / 633,486, filed Dec. 6, 2004, Dec. 6, 2004 Application No. 60 / 633,453, filed on Dec. 9, Application No. 60 / 634,627, filed Dec. 9, 2004, Application No. 60 / 634,739, filed Dec. 9, 2004 No. 60 / 647,684, filed Jan. 26, 2005, No. 60 / 648,746, filed Jan. 31, 2005, filed Feb. 15, 2005 Application 60 / No. 53,372, Application No. 60 / 653,663 filed on Feb. 16, 2005, No. 60 / 653,669 filed Feb. 16, 2005, Feb. 16, 2005. Application 60 / 653,899, application 60 / 653,679 filed February 16, 2005, application 60 / 653,847 filed February 16, 2005, Application No. 60 / 654,379 filed on Feb. 17, 2005, Application No. 60 / 654,368 filed on Feb. 18, 2005, Application No. filed on Feb. 18, 2005 No. 60 / 654,326, Application No. 60 / 654,196 filed on Feb. 18, 2005, No. 60 / 655,279 filed Feb. 22, 2005, Feb. 22, 2005 Filed on the day Application No. 60 / 655,280, Application No. 60 / 655,987 filed on February 22, 2005, Application No. 60 / 655,697 filed on February 22, 2005, 2005 Application 60 / 655,281 filed February 22, Application 60 / 657,309 filed February 28, 2005, Application 60/657 filed July 24, 2006 No. 833,131, Application No. 60 / 843,362 filed on September 8, 2006, Application No. 60 / 844,894 filed on September 15, 2006, September 15, 2006 Application 60 / 844,893 filed, Application 60 / 845,604 filed September 18, 2006, and Application 60 / 910,438 filed April 5, 2007. .

(Technical field)
The described techniques are directed to the field of interaction with rendered documents, and more specifically to operate in response to information captured from rendered documents.

  Paper documents have a permanent appeal as can be seen by the proliferation of paper documents in the computer age. Printing and publishing paper documents is not as easy as it is today. Even though electronic documents are easier to duplicate, send, search, and edit, paper documents are dominant.

  Considering the prevalence of paper documents and the benefits of electronic documents, it is useful to combine the benefits of both.

It is a data flow figure showing the flow of information in one embodiment of a core system. FIG. 2 is a component diagram of components included in an exemplary implementation of the system in the context of a general operating environment. It is a block diagram which shows one Embodiment of a scanner. 1 is a system diagram illustrating a typical environment in which a system operates. FIG. 6 is a flow diagram illustrating the steps normally performed by the system to perform an action in response to a user keyword capture. It is a table figure which shows the sample content of a keyword operation | movement table. It is a table figure which shows the sample content of the document action map with respect to a specific document. FIG. 5 is a flow diagram illustrating the steps typically performed by the system to perform an action in response to user-ingested material not associated with a keyword. FIG. 5 is a flow diagram illustrating the steps typically performed by the system to present an advertisement in response to document identification.

Overview Software for triggering actions such as advertisements in response to optically or acoustically capturing keywords from a rendered document, or in response to identifying a document based on the captured keywords The hardware system will be described (“system”). As used herein, a keyword means one or more words, icons, symbols, or images. While the terms “word” and “plural word” are often used in this application, in some embodiments, icons, symbols, or images may be used. A keyword as used herein also refers to a phrase composed of one or more adjacent symbols. The keywords used herein include words relating to topics or subjects addressed in the rendered document, or a portion of the rendered document identified in response to the capture.

  Keywords are considered “overloaded”, that is, keywords have certain associated meanings or actions beyond the general (eg, visual) meaning for the user, such as text or symbols. In some embodiments, the association between keywords and meanings or actions is set by a markup process or data. In some embodiments, the association between a keyword or document and a meaning or action is known to the system when it is captured or identified. In some embodiments, the association between keywords or documents and meanings or actions is established after capture or identification has taken place. Semantics or actions are optionally in the stage of displaying relevant annotations or relevant content, some keywords that can be seen on the display have special meaning or can be implemented immediately (e.g. These can include visual indications on a dynamic display that responds to mouse clicks and the system presents a menu of available choices when the cursor rests on a keyword.

  In some embodiments, the system uses the content of the document to identify the document and to trigger and select advertisements presented to the user. In some cases, the system can parse the document and associate the content of the document with one or more keywords. In some cases, the system selects an advertisement (action) based on the content of the entire document. In some cases, the system selects an advertisement based on a portion of the document. In some cases, the system selects an advertisement based on the content of the document that is not used when identifying the document.

Part 1-Introduction 1. System Properties For every paper document that has a corresponding electronic document, there is a discrete amount of information in the paper document that can identify the corresponding electronic document. In some embodiments, the system uses a sample of text captured from a paper document, for example using a handheld scanner, to identify and locate the corresponding electronic document of the document. In most cases, the amount of text required by a function is very small and a few words of text from a document can often serve as an identifier for a paper document and a link to its corresponding electronic document. In addition, the system can use the few words to identify not only the document but also the location within the document.

  Thus, a paper document and its corresponding digital document can be associated in many useful ways using the system described herein.

1.1. Future Overview When the system associates a portion of text in a paper document with a specific digital entity that has been set, the system can build a huge amount of functionality on that association.

  Many paper documents have corresponding electronic documents that can be accessed on the World Wide Web or from some other online database or document corpus, or can be made accessible, for example, upon payment of fees or subscription fees Has increased. At the simplest level, when a user scans a few words of a paper document, the system retrieves or displays this electronic document or part thereof, emails someone, purchases, prints, or Can be posted on a web page. As a further example, by scanning a few words of a book that a person is reading during breakfast, the audio-book version in the person's car is read from when the person started driving to work. Alternatively, the replacement ordering process can be initiated by scanning the serial number of the printer cartridge.

  The system implements these and many other examples of “paper / digital integration” without the need to change the current process of writing, printing, and publishing documents, and such conventional rendering. Bring a whole new layer of digital functionality to your documents.

1.2. Terminology The general use of the system begins with scanning text from a paper document using an optical scanner, but note that other capture methods from other types of documents are equally applicable. Is important. Thus, the system may have been described as scanning or capturing text from a rendered document, where these terms are defined as follows:

  The rendered document is a printed document or a document displayed on a display or monitor. A document that can be perceived by humans in either a permanent form or a temporary display.

  Scanning or capturing is a systematic inspection process for obtaining information from a rendered document. This process can include optical capture using a scanner or camera (eg, a cell phone camera), or it can include reading from a document to an audio capture device or typing on a keypad or keyboard. In some embodiments, optical capture is performed by determining what data is optically visible (eg, rendered) to the user on all or part of the dynamic display. This optical capture of visible data can be achieved by analyzing the display's storage buffer (eg, by performing optical character recognition or other image analysis on the image stored in the display buffer) for dynamic display. Generating and / or generating display data by intercepting and analyzing changes made to the display (eg, by analyzing new data written to the display when the user changes the visible data by scrolling through the window) Achieved by requesting information about the display data from the application or operating system component responsible for accessing the display data, or otherwise determining what keywords and other content the user is viewing can doIn some embodiments, text rendered on a dynamic display is converted into a series of signatures (eg, by taking a contiguous group of 100 characters in a sliding window) and querying the database to identify these It is determined whether there is an associated action or content for any of the signatures. See Section 15 for scan and capture examples.

2. System Introduction This section describes some of the devices, processes, and systems that make up the system for paper / digital integration. In various embodiments, the system builds various services and applications on this basic core that provides basic functionality.

2.1. Process FIG. 1 is a data flow diagram illustrating the flow of information in one embodiment of a core system. Other embodiments may not use all of the steps or elements shown herein, but others use more.

  Text from the rendered document is typically captured (100) in optical form by an optical scanner or in speech form by a voice recorder, and this image or sound data is then used, for example, to remove artifacts of the capture process or signal Processed to improve the noise to noise ratio (102). A recognition process 104, such as OCR, speech recognition, or autocorrelation, then optionally converts the data into one or more signatures that in some embodiments consist of text, text offsets, or other symbols. Convert. Alternatively, the system performs other forms of extracting a document signature from the rendered document. A signature represents a set of textual representation formats that can be implemented in some embodiments. This process may be affected by feedback from other stages, for example, if the search process and context analysis 110 identifies several candidate documents that may be ingested, Possible interpretations may be narrowed.

  The post-processing 106 stage can take the output of the recognition process and perform other operations that can be filtered or useful. Depending on the embodiment being implemented, this stage may be taken immediately without reference to a later stage, such as when a phrase or symbol containing essentially enough information to convey the user's intention is captured. Some direct actions 107 can be made speculative. In these cases, there is no need to reference the corresponding digital document or to be known to the system.

  However, typically the next stage will constitute a query 108 or set of queries for use in the search. Some aspects of query construction may depend on the search process used, and thus cannot be performed until the next stage, but in advance such as removal of clearly misrecognized characters or inappropriate characters. There will be several operations that can be performed.

  The one or more queries are then passed to the search and context analysis stage 110. Here, the system optionally attempts to identify the document in which the original data was captured. To do this, the system typically uses a search index and search engine 112, knowledge 114 about the user, and knowledge 116 about the user's context or the context in which it was captured. The search engine 112 can utilize and / or index information about, among other things, rendering documents, about these corresponding digital documents, and about documents with web (Internet presence). The search engine 112 can write to and read from many of these sources and, as described above, for example, for the language, font, rendering and possible next word based on knowledge of the candidate document. Can be provided to the recognition system 104 to provide information to other stages of the process.

  In some situations, the next stage will retrieve 120 copies of the identified document or documents. The document source 124 can be directly accessible, for example, from a local filing system or database or web server, or contacted through several access services 122 that can perform authentication, security, or payment. Other services can be provided, such as conversion of documents to a desired format.

  Applications of the system can take advantage of additional features or associations between data and some or all of the documents. For example, the advertising application discussed in Section 10.4 may use an association with a particular advertising message or a portion of a subject's document. This additional related function or data can be thought of as one or more overlays on the document and is referred to herein as “markup”. The next stage of process 130 is then identifying any markup for the captured data. Such markup can be provided by the user, drafter, or issuer, or any other party, can be directly accessible from some source 132, or generated by some service 134. can do. In various embodiments, the markup can be applied to the rendered document and / or the corresponding digital document of the rendered document, or in association with either or both groups of these documents.

  Finally, some actions can be taken as a result of the earlier stages (140). These can be default actions such as simply recording the information found, or can depend on data or documents, or they can be derived from markup analysis. In some cases, the action simply passes the data to another system. In some cases, various possible actions suitable for capture at a particular point in the rendered document are displayed as menus on the local display 332, the computer display 212 or the associated display on the mobile phone or PDA display 216, for example, as a user. Will be presented. If the user does not respond to the menu, a default action can be performed.

2.2. Components FIG. 2 is a component diagram of components included in a typical implementation of a system in the context of a typical operating environment. As shown, the operating environment includes one or more optical scan capture devices 202 or audio capture devices 204. In some embodiments, the same device performs both functions. Each capture device, such as a computer 212 and a mobile station 216 (eg, a mobile phone or PDA) via a network 220 that can communicate using either a direct wiring or wireless connection, or using a wired or wireless connection, etc. The network 220 typically includes a radio base station 214 that can communicate with other parts of the system. In some embodiments, the capture device is integrated into the mobile station and optionally shares some of the audio and / or optical components used in the device for audio communication and imaging.

  Computer 212 may include a memory that contains computer-executable instructions for processing instructions from scan devices 202 and 204. As an example, the instructions may include an identifier (such as a serial number of the scan device 202/204 or an identifier that partially or uniquely identifies the scanner user), scan context information (eg, scan time, scan location, etc. ) And / or scan information (such as text strings) used to uniquely identify the scanned document. In other embodiments, the operating environment may include more or fewer components.

  Search engine 232, document source 234, user account service 236, markup service 238, and other network services 239 are also available on network 220. The network 220 can be a corporate intranet, public internet, cellular network, or some other network, or any of the interconnections described above.

  Regardless of how the devices are coupled to each other, they can all be operable according to known commerce and communication protocols (eg, Internet Protocol (IP)). In various embodiments, the functions and features of the scanning device 202, the computer 212, and the mobile station 216 can be fully or partially integrated into one device. Thus, the terms scanning device, computer, and mobile station can refer to the same device depending on whether the device incorporates the functions or capabilities of scanning device 202, computer 212, and mobile station 216. Further, some or all of the functionality of search engine 232, document source 234, user account service 236, markup service 238, and other network services 239 may be on any of the devices and / or other devices not shown. Can be implemented.

2.3. Capture Device As described above, a capture device can capture text using an optical scanner that captures image data from a rendered document, or using an audio recording device or other method that captures user text readings. it can. Some embodiments of the capture device can also capture images, graphical symbols and icons, etc., including machine readable codes such as barcodes. The device can be quite simple and can consist of as many converters, several storage devices and data interfaces as other functions resident elsewhere in the system, or even a full-featured device It can be. As an illustration, this section describes a device that is based around the optical scanner and that has a reasonable number of functions.

  A scanner is a known device that captures and digitizes an image. The first scanner, a byproduct of the copier industry, was a relatively large device that captured entire document pages at once. Recently, portable optical scanners such as pen-type handheld devices have been introduced with convenient dimensions.

  In some embodiments, a portable scanner is used to scan text, graphics, or symbols from the rendered document. Portable scanners have scanning elements that capture text, symbols, graphics, etc. from a rendered document. In some embodiments, in addition to documents printed on paper, rendered documents include documents displayed on a screen such as a CRT monitor or LCD display.

  FIG. 3 is a block diagram of an embodiment of the scanner 302. The scanner 302 scans information from the rendered document and converts it into machine compatible data and an optical scan head 308, and an optical path 306, a lens that normally conveys the image from the rendered document to the scan head, Includes aperture or image conduit. The scan head 308 can incorporate a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) imaging device, or another type of photosensor.

  The microphone 310 and associated circuitry converts ambient sounds (including spoken language) into machine compatible signals, and other input functions exist in the form of other tactile sensors 314 such as buttons, scroll wheels, or touch pads. To do.

  Feedback to the user can be performed via a visual display or indicator light 332, via a loudspeaker or other audio transducer 334, and via a vibration module 336.

  The scanner 302 includes logic circuitry 326 for interacting with various other components and can process received signals into different formats and / or interpretation information. The logic circuit 326 may be operable to read and write data and program instructions stored in associated storage devices 330 such as RAM, ROM, flash, or other suitable memory. The logic circuit 326 can read the time signal from the clock unit 328. The scanner 302 also includes an interface 316 for communicating scanned information and other signals to the network and / or associated computing devices. In some embodiments, the scanner 302 can have an on-board power source 332. In other embodiments, the scanner 302 can be powered from a tether connection to another device, such as a universal serial bus (USB) connection.

  As an example of one application of the scanner 302, a reader can scan some text from a newspaper article with the scanner 302. The text is scanned as a bitmap image via the scan head 308. Logic circuit 326 stores the bitmap image in memory 330 along with the associated timestamp read from clock unit 328. The logic circuit 326 can also perform optical character recognition (OCR) or other post-scan processing on the bitmap image and convert it to text. The logic circuit 326 optionally extracts a signature from the image, for example, by performing a convolution-like process to locate the occurrence of a character, symbol, or object iteration, between these iteration elements. The spacing or number of other characters, symbols, or objects can be determined. The reader can then upload the bitmap image (or text or other signature, if post-scan processing is performed by logic circuit 326) to the associated computer via interface 316.

  As an example of another application of the scanner 302, a reader can capture some text from an article as an audio file by using the microphone 310 as an acoustic capture port. The logic circuit 326 stores the audio file in the memory 328. Logic circuit 326 can also perform voice recognition or other post-scan processing on the audio file and convert it to text. As described above, the reader can then upload the audio file (or text generated by post-scan processing performed by logic 326) to the associated computer via interface 316.

Part 2-Overview of the Core System Domain As paper-digital integration becomes more common, existing ones that can be modified to make better use of this integration or to be able to be implemented more effectively There are many aspects of this technology. This section clarifies some of these issues.

3. Search Searching a document corpus, and even a large corpus such as the World Wide Web, has become commonplace for ordinary users who construct search queries that are sent to a search engine using a keyboard. This section and the next section describe aspects of both the construction of queries generated by ingestion from rendered documents and the search engine that processes such queries.

3.1. Types as Scan / Speak / Search Queries The use of the described system will typically result in several words from the rendered document using any of several methods, including those mentioned in section 1.2 above. It starts with being taken in. End-to-end feedback is present in the system so that the recognition process can be extended using a document corpus if the input requires some interpretation to be converted to text, for example in the case of OCR or speech input Can do. End-to-end feedback provides an approximation of recognition or interpretation, identifies a set of one or more candidate matching documents, and then uses the information from possible matches in the candidate documents to recognize or interpret It can be applied by further improvement or limitation. Candidate documents can be weighted according to their possible relevance (eg, based on the number of other users who scanned these documents, or their popularity on the Internet), and these weights are It can be applied in an iterative recognition process.

3.2. Short phrase search When the relative position of a few words is known, the selectivity of a search query based on a few words is greatly enhanced so that the system must capture the location of the text in the corpus The text is only a small amount. Usually, the input text is a sequence of consecutive words such as a short phrase.

3.2.1. Finding documents and in-document locations from short captures In addition to locating the document from which the phrase originates, the system can identify the location in the document and take action based on this knowledge.

3.2.2. Other Methods of Finding Locations The system can also employ other methods of finding documents and locations, such as using watermarks or other special markings in rendered documents.

3.3. Including other elements in the search query In addition to the captured text, other elements (ie information about the user identity, profile, and context) include the capture time, the user's identity, and the geographic location, Part of the search query can be formed, such as knowledge about habits and recent activity.

  Document identities and other information related to previous captures can form part of the search query, especially if they are very recent.

  The user's identity can be determined from a unique identifier associated with the capture device and / or biometric or other supplemental information (speaking, fingerprint, etc.).

3.4. Knowledge of the nature of unreliability in search queries (such as OCR errors)
The search query can be constructed taking into account the types of errors that can occur with the particular capture method used. One example of this is the display of suspicious errors in the recognition of certain characters, in which case the search engine can treat these characters as wildcards or assign them a low priority.

3.5. Local caching / offline use of indexes for performance The capture device may not be able to communicate with the search engine or corpus during data capture. For this reason, information useful for offline use of the device can be downloaded to the device in advance or to some entity with which the device can communicate. In some cases, all or a substantial portion of the index associated with the corpus can be downloaded. This topic will be discussed further in Section 15.3.

3.6. Queries that can be recorded in any form and executed later This preloaded if there is a high probability of delay or cost associated with conveying queries or receiving results Information can improve local device performance, reduce communication costs, and provide more useful and timely user feedback.

  In situations where communication is not available (local device is “offline”), the query is saved and can be sent to the rest of the system, such as when communication is restored.

  In these cases, it may be important to send a time stamp with each query. Capture time can be an important factor in query interpretation. For example, Section 13.1 discusses the importance of capture time in relation to early capture. It is important to note that the capture time is not always the same as when the query was executed.

3.7. Parallel search For performance reasons, multiple queries can be initiated sequentially or in parallel in response to a single ingestion. In response to a single capture, several queries can be sent, for example when a new word is added to the capture, or to query multiple search engines in parallel.

  For example, in some embodiments, the system sends the query to a special index of the current document, a search engine on the local machine, a search engine on the corporate network, and a remote search engine on the Internet.

  Certain search results can be given higher priority than other searches.

  A response to a given query may indicate that other pending queries are excessive, and these can be canceled before completion.

4). Paper and search engines Traditional search engines that handle online queries often also want to handle queries that originate from rendered documents. Conventional search engines can be enhanced or modified in several ways to make them more suitable for use with the described system.

  The search engine and / or other components of the system can create and maintain indexes with different or additional features. The system can modify incoming paper-derived queries or change the way queries are handled in the resulting search, so these paper-derived queries have been typed into web browsers and other sources It can be distinguished from that resulting from the query. The system may take different actions when compared to those from other sources or provide different options when results are returned by a paper-derived search. Each of these approaches is described below.

4.1. Indexing In many cases, paper-derived or conventional queries can be used to search for the same index, but the index can be enhanced for use in various ways in current systems.

4.1.1. Knowledge of the paper format Additional fields can be added to the index to aid in paper-based searches.

Index entry indicating document availability in paper format The first example is a field indicating that a document is known to exist or be distributed in paper format. The system can give high priority to such documents if the query comes from paper.

Knowledge of paper form popularity In this example, statistical data (eg, amount of scan activity, issuer or other) about the popularity of paper documents (and optionally about sub-regions within these documents) The number of publications provided by the source) is used to give these documents high priority, increase the priority of corresponding digital documents (eg, for browser-based queries or web search), etc. .

Render Format Knowledge Another important example may be a record of information about the layout of a particular rendering of a document.

  For example, in a particular edition of the book, the index may include information about where line breaks and page breaks occur, which font was used, or any exceptional capitalization.

  The index can also include information about the proximity of other items on the page, such as images, text boxes, tables, and advertisements.

Using semantic information in the original Finally, it can be inferred from source markup that a specific part of the text refers to an item being marketed or that a paragraph contains program code, but in a paper document Semantic information that is not obvious can also be recorded in the index.

4.1.2. Indexing in knowledge of capture methods A second factor that can modify the nature of the index is knowledge of the type of capture that is likely to be used. A search initiated by an optical scan can be advantageous if the index takes into account characters that are likely to be confused in the OCR process or if it contains some knowledge of the fonts used in the document. Similarly, if the query is provided by speech recognition, an index based on similar phonemes can be searched much more efficiently. An additional factor that can affect the use of the index in the model described is the importance of iterative feedback during the recognition process. If the search engine can provide feedback from the index when the text is captured, the search engine can greatly increase the accuracy of the capture.

Indexing with offsets If the index is likely to be searched using the offset-based / autocorrelation OCR method described in Section 9, in some embodiments, the system may include an appropriate offset or signature within the index. Store information.

4.1.3. Multiple indexes Finally, in the system described, it can be common to perform a search on many indexes. The index can be maintained on multiple machines in the corporate network. A partial index can be downloaded to the capture device or a machine near the capture device. A separate index can be created for users or groups of users with specific interests, habits, or permissions. An index can exist for each file system, each directory, and each file on the user's hard disk. The index is issued and used by users and systems. Therefore, it is important to construct an index that can be efficiently distributed, updated, merged, and split.

4.2. Handling of queries 4.2.1. Recognition of capture from paper The search engine can take different actions when it recognizes that the search query is from a paper document. The engine can handle queries in a manner that is more tolerant to, for example, the types of errors that are likely to appear in certain capture methods.

  The search engine can infer this from some indicator included in the query (eg, a flag indicating the nature of the ingest), or it can be inferred from the query itself (eg, an error or error specific to the OCR process) Uncertainty can be recognized).

  Alternatively, queries from the capture device can reach the engine by a different channel or port or connection type than those from other sources and can thus be distinguished. For example, in some embodiments of the system, queries are routed to a search engine via a dedicated gateway. Thus, the search engine knows that all queries that pass through the dedicated gateway are from paper documents.

4.2.2. Use of Context Section 13 below describes a variety of different factors that are outside the captured text itself but can significantly help in document identification. These include things such as the history of recent scans, a particular user's long-term reading habits, the user's geographic location, and the user's recent use of a particular electronic document. Such factors are referred to herein as “contexts”.

  Part of the context is handled by the search engine itself and is reflected in the search results. For example, a search engine can track a user's scan history, and can cross-reference this scan history with conventional keyboard-based queries. In such cases, the search engine retains and uses more state information for each individual user than most traditional search engines do, and each interaction with the search engine is common across multiple searches and today It can be thought of as extending over a longer period of time than is desired.

  Some of the context can be sent to the search engine in a search query (section 3.3) and in some cases can be stored in the engine to work to some extent in future queries. Finally, some of the context is best handled elsewhere, resulting in a filter or secondary search applied to the results from the search engine.

Data Stream Input for Search An important input to the search process is the broad context regarding how the user community interacts with the rendered version of the document, for example which document is most widely read by who. There are similarities for web searches that return the most frequently linked pages or the most frequently selected pages from past search results. See Sections 13.4 and 14.2 for further discussion on this topic.

4.2.3. Document Subregion The described system can provide and use not only information about the entire document, but also information about subregions of the document down to individual words. Many existing search engines are focused solely on locating documents or files that are relevant to a particular query. A search engine that works on higher granularity and can identify locations within a document will provide significant benefits to the described system.

4.3. Returning results The search engine can use some of the additional information it currently maintains to influence the returned results.

  The system can also return only certain documents accessed by the user as a result of having a paper copy (section 7.4).

  The search engine can also provide new actions or options appropriate to the described system besides simple text retrieval.

5. Markup, Annotation, and Metadata In addition to performing the capture-search-retrieve process, the described system also associates additional functionality with a document, specifically a specific location or segment of text within the document. This additional functionality is often, but not limited to, associated with the rendered document by being associated with its corresponding electronic document. As one example, a hyperlink in a web page may have the same function when a printout of that web page is scanned. In some cases, this functionality is not defined in the electronic document, but is stored or generated elsewhere.

  This layer of additional functionality is referred to herein as “markup”.

5.1. Overlay, static and dynamic One approach to markup is what can be considered as an “overlay” for a document, providing additional information about the document or part of the document and specifying these related actions. Markups can include human-readable content, but are often invisible to the user and / or intended for machine applications. Examples include options that are displayed in a pop-up menu on a nearby display when the user captures text from a particular area of the rendered document, and audio samples that show the pronunciation of a particular phrase.

5.1.1. Multiple layers provided from multiple sources in some cases Any document can have multiple overlays at the same time, and these can be sourced from various locations. The markup data can be created or supplied by the author of the document, by the user, or by some other party.

  Markup data can be attached or embedded in an electronic document. This can be found at a conventional location (eg, at the same location as the document but with a different filename suffix). The markup data can be included in the search results of the query that locates the original document, or can be found by separate queries to the same or another search engine. The markup data can be found using the original capture text and other capture information or context information, or can be found using information already guessed about the document and capture location. Markup data can be found at a location specified in the document even if the markup itself is not included in the document.

  Markup can be highly static and document-specific, similar to the way links on traditional html web pages are often embedded as static data in html documents, Up can also be dynamically generated and / or applied to multiple documents. An example of dynamic markup is information attached to a document that includes the latest stock prices of the companies mentioned in the document. A widely applied example of markup is translation information that is automatically available in multiple documents or document sections in a particular language.

5.1.2. Personal “plug-in” layer users can also install markup data or subscribe to specific sources of markup data, thereby personalizing system responses to specific captures.

5.2. Keywords and phrases, trademarks, and logos Some elements in a document may have specific “markup” or related functionality based on unique features other than location in the specific document. Examples include special marks that are printed in the document for pure scanning purposes, as well as logos and trademarks that can link the user to further information about the organization of interest. The same applies to “keywords” or “key phrases” in text. An organization can register a specific phrase to be associated with, or want to be associated with, and attach to it a special-purpose markup that is available wherever the phrase is scanned.

  Every word, phrase, etc. can have an associated markup. For example, the system can add a specific item to a pop-up menu whenever the user captures the word “book” or the title of a book or a topic related to a book (eg, a link to an online book store). ). In some embodiments of the system, the corresponding digital document or index is queried to determine whether a capture was made near the word “book” or the title of the book, or a topic related to the book, and the operation of the system. Determine if it has been modified according to this proximity to the keyword element. It should be noted that in the foregoing embodiment, markup allows data captured from non-commercial text or documents to trigger a commercial transaction.

5.3. User-provided content 5.3.1. User comments and annotations, including multimedia Annotations are another type of electronic information that can be associated with a document. For example, the user can attach an audio file of the user's view of a particular document for later retrieval as a voice annotation. As another example of multimedia annotation, the user can attach a photo of the location mentioned in the document. Users generally provide annotations for documents, but the system can associate annotations from other sources (eg, other users in a workgroup can share the annotations).

5.3.2. Notes from Proofread An important example of user source markup is the annotation of paper documents as part of the proofreading, editing or review process.

5.4. Third Party Content As mentioned above, markup data can often be provided by a third party, such as other readers of a document. Online discussions and reviews, such as community management information about specific tasks, translations and explanations by volunteer contributions, are preferred embodiments.

  Another example of third party markup is that provided by an advertiser.

5.5. Dynamic markup based on other users' data streams Multiple or all users of the system generate markup based on community activity and interests by analyzing data captured from documents be able to. As an example, it could actually be an online book store that creates markups or annotations that indicate to the user that “the person who enjoyed this book also enjoyed ...”. The markup can be less anonymous and can indicate to the user who in the user's contact list has recently read this document. Other examples of data stream analysis are included in section 14.

5.6. Markup based on external events and data sources Markups are often based on external events and data sources, such as input from corporate databases, information from the public Internet, or statistics collected by the local operating system .

  Data sources can also be more local, and in particular can provide information about the user's context, ie the user's identity, location, and activity. For example, the system can provide a markup layer that communicates with the user's cell phone and gives the user the option to send the document to the person the user recently spoke on the phone.

6). Authentication, personalization, and security In many cases, a user's identity is known. In some cases this will be an “anonymous identity” where the user is identified only by the serial number of the capture device, for example. However, in general, the system has much more detailed knowledge of the user and can be used to personalize the system and to perform activities and transactions with the user's name. It is expected to be.

6.1. User history and "life library"
One of the simplest and most useful functions that the system can perform is any additions to the capture, including text captured by the user, details of every document found, location within that document, and any action taken as a result. Maintaining a record of the information about the user.

  This stored history is beneficial to both the user and the system.

6.1.1. About the User The user can be presented with a “life library” that is a record of everything the user has read and captured. This can be solely for personal interest, but can be used in a library by a researcher collecting material for a reference for a user's next paper, for example.

  In some situations, a user may wish to make the library public, such as by publishing the library on the web in a manner similar to a weblog, which allows others to read and be interested in Will be able to see.

  Finally, in situations where the user captures some text and the system cannot respond immediately to the capture (eg, because an electronic version of the document is not yet available), the capture can be stored in the library and automatically Alternatively, it can be processed later in response to a user request. Users can also subscribe to new markup services and apply them to previously captured scans.

6.1.2. About the system Recording the user's past captures is also useful for the system. Many aspects of system operation can be enhanced by knowing the user's reading habits and history. In the simplest example, any scan performed by a user is likely to have originated from a document that the user recently scanned, especially if the last scan was within the last few minutes, Very likely from the same document. Similarly, it is likely that the document has been read from the beginning to the end. Thus, for English documents, later scans are likely to occur much lower in the document. Such factors can help establish the location of the capture when the system is ambiguous and can also reduce the amount of text that needs to be captured.

6.2. Scanner as a payment, identity, and authentication device The capture process generally begins with some type of device, usually an optical scanner or voice recorder, so this device identifies the user as a key to allow certain actions. Can be used.

6.2.1. Associating a scanner with a phone or other account The device can be built into the cell phone or associated with the cell phone account in some other way. For example, a scanner can be associated with a mobile phone account by inserting a SIM card associated with the account into the scanner. Similarly, the device can be embedded in a credit card or other payment card, or can have the functionality to which such a card is connected. Thus, the device can be used as a payment token and can initiate a financial transaction upon capture from a rendered document.

6.2.2. Using Scanner Input for Authentication A scanner can also be associated with a particular user or account through the process of scanning any token, symbol, or text associated with that particular user or account. Furthermore, the scanner can be used for biometric identification, for example by scanning the user's fingerprint. For voice-based capture devices, the system can identify the user by matching the user's voiceprint or by requesting the user to speak a particular password or phrase.

  For example, if a user scans a citation from a book and is offered an option to purchase the book from an online retailer, the user can select this option and then scan the user's fingerprint to confirm the transaction Prompted.

  See also sections 15.5 and 15.6.

6.2.3. Secure Scan Device When a capture device is used to identify and authenticate a user and initiate a transaction on behalf of the user, it is important that the communication between the device and the rest of the system is secure. It is also important to defend against situations where another device impersonates a scanner and so-called “man-in-the-middle” attacks where communication between the device and other components is intercepted.

  Techniques for providing such security are well understood in the art, and in various embodiments, the devices and other hardware and software in the system may use such techniques. Configured to be implemented.

7). Publishing Models and Elements An advantage of the described system is that it does not require changes to the traditional process of creating, printing, or publishing documents to obtain many of the benefits of the system. However, there are reasons that document creators or publishers (hereinafter simply referred to as “issuers”) may wish to create functionality to support the described system.

  This section is primarily concerned with the published document itself. For information on other relevant commercial transactions, such as advertisements, see Section 10 entitled “P-Commerce”.

7.1. Electronic and Printed Documents The system allows printed documents to have an associated electronic presence. Traditionally, publishers often ship CD-ROMs with books that contain additional digital information, tutorial movies and other multimedia data, sample code or documents, or additional related materials. In addition, some publishers update not only such materials but also post-publishing errata, additional comments, updated related materials, additional sources of references and related data, and translations into other languages. Maintain a website associated with a specific publication that provides information that it can do. Online forums allow readers to contribute comments about the publication.

  The described system allows such materials to be more closely related to rendered documents than ever before, making their discovery and interaction with them much easier for the user. Become. By capturing a portion of text from a document, the system can automatically tie the user to digital material associated with the document, and more specifically, associated with a particular portion of the document. Similarly, users are tied to annotations and interpretation by online communities or other readers discussing sections of text. In the past, such information typically had to be found by searching for specific page numbers or chapters.

  This exemplary application is in the field of academic books (section 17.5).

7.2. "Subscription" for printed documents
Some publishers may have mailing lists that readers can subscribe to if they want to be notified of new related matters or when a new edition of a book is published. In the described system, users can easily register interest in a particular document or part of a document, even before the publisher considers providing any such functionality. Can do. The reader's interest is provided to the publisher and affects decisions about when and where to provide updates, more information, new editions, or entirely new publications on topics found to be of interest to existing books There is.

7.3. Many aspects of print mark systems that have special meaning or contain special data are made possible simply through the use of text already present in the document. However, if a document is created with the knowledge that it can be used with the system, additional functionality can be added by printing additional information in the form of special marks, which can be used to create text or required Actions can be more closely identified, or document interaction with the system can be enhanced. The simplest and most important example is an indication to the reader that the document is clearly accessible through the system. For example, a special icon can be used to indicate that the document has an associated online discussion forum.

  Such symbols can be purely intended for the reader, or can be recognized by the system when scanned and used to initiate some action. Enough data can be encoded in symbols to identify more than just symbols, for example, information about the location of documents, editions, and symbols can be stored, these Can be recognized and read by the system.

7.4. Authorization through possession of a paper document Ownership or access to a printed document can give the user certain privileges, such as an electronic copy of the document or access to additional material. Using the described system, such privileges can be granted simply as a result of a user scanning a portion of text from a document or a specially printed symbol. If the system needs to verify that the user owned the entire document, the system prompts the user to scan a specific item or phrase from a specific page, for example, “second line on page 46”. be able to.

7.5. Expired Documents If the printed document is a gateway to additional materials and functionality, access to such functions can also be timed. After the expiration date, the user may be required to pay a fee or obtain a newer version of the document to access the article again. Of course, paper documents can still be used, but some of their enhanced electronic functionality will be lost. For example, because the publisher is profitable by receiving a fee for access to electronic materials, or by asking the user to purchase new editions as appropriate, or related to older versions of printed documents that are still in circulation This can be desirable because there are disadvantages to doing. A coupon is an example of a type of commercial document that can have an expiration date.

7.6. Popularity analysis and publication decisions Section 10.5 discusses the use of system statistics to influence author rewards and advertising pricing.

  In some embodiments, the system infers the popularity of the publication from activity in the associated electronic community as well as from the use of paper documents. These factors can help issuers make decisions about what will be issued in the future. For example, if an existing book chapter turns out to be very popular, it may be worth deploying in a separate publication.

8). Document Access Service An important aspect of the described system is the ability to allow a user with access to a rendered copy of a document to access an electronic version of the document. In some cases, documents are freely available on public or private networks accessible to the user. The system uses the captured text to identify, locate, and retrieve the document, possibly displaying it on the user's screen, or depositing it in the user's email inbox.

  In some cases, even though the document is available in electronic form, the user may not be able to access it for various reasons. In some cases, there may not be enough connectivity to retrieve the document, the user may not be entitled to retrieve the document, and there may be costs associated with accessing the document. Or, the document may have been collected and in some cases replaced with a new version. The system typically provides feedback to the user about these situations.

  As described in Section 7.4, the degree or nature of access granted to a particular user may vary if it is known that the user already has access to a printed copy of the document. .

8.1. Authenticated document access Access to documents can be restricted to designated users or users who meet certain conditions, or in certain circumstances, such as when the user is connected to a secure network. Can only be available. Section 6 describes some of the ways in which user and scanner credentials can be established.

8.2. Document Purchase—Copyright Owner Reward Documents that are not freely available to the general public can be made accessible by paying a fee, often as a reward to the issuer or copyright owner. The system can directly implement the payment function, or can utilize other payment methods associated with the user, including those described in Section 6.2.

8.3. Document Escrow and Active Search Electronic documents are often temporary and digital source versions of rendered documents are currently available but may not be accessible in the future. The system can retrieve and store an existing version on behalf of the user, even if the user does not request it, thus ensuring its availability if the user requests it in the future. This also allows the system to be used to search, for example, as part of the process of identifying future captures.

  In the event that payment is required to access a document, a reliable "document escrow" service can retrieve the document on behalf of the user, such as by paying a small fee, and the user requests the document from the service In some cases, the copyright owner is guaranteed to be fully paid in the future.

  Variations of the present subject matter can be implemented if the document is not available in electronic form at the time of capture. The user can authorize the service to submit a document request or pay for the document on behalf of the user when the electronic document becomes available at a later date.

8.4. Association with other subscriptions and accounts. In some cases, payments can be abandoned, reduced, or repaid based on the user's existing association with another account or subscription. For example, a subscriber to a printed version of a newspaper can be automatically authorized to retrieve an electronic version.

  In other cases, the association cannot be made completely direct and the user can be based on an account set up by the employer or based on a scan of a printed copy owned by a friend who is a subscriber. Access can be granted.

8.5. Scanning and printing as an alternative to photocopying The process of taking text from a paper document, identifying an electronic original, and printing that original or part of the original associated with the capture is an alternative to conventional photocopying, with many advantages. Form. That is,
Paper documents do not need to be in the same location as the final printout, and need not be present at the same time in any case. Avoid wear and damage caused by the photocopying process, especially old and vulnerable valuable documents.・ Usually the copy quality is much higher ・ You can keep a record of which documents or parts of documents are most frequently copied ・ Pay the copyright owner as part of the process・ Unauthorized copying can be prohibited

8.6. A worthy original position by photocopying If the document is particularly valuable, as in the case of a legal deed or document of historical or other specific significance, it is usually often over many years. You can work with copies of these documents and keep your originals in a secure location.

  The described system can be coupled, for example, to a database that records the location of the original document in an archive warehouse, making it easy for anyone to access the copy to locate the archived original paper document.

9. Text Recognition Technology Traditionally, optical character recognition (OCR) technology has focused on images containing large amounts of text from, for example, flatbed scanners that capture entire pages. OCR techniques often require considerable training and correction by the user to produce useful text. OCR technology often requires significant processing power on the machine performing OCR, and when many systems use dictionaries, they are generally expected to operate with a virtually unlimited vocabulary.

  All of the above conventional features can be improved in the described system.

  Although this section focuses on OCR, many of the issues considered are directly mapped to other recognition technologies, particularly speech recognition. As described in Section 3.1, the process of capturing from paper can be accomplished by reading the text aloud to a device where the user captures the sound. One skilled in the art will appreciate that the principles discussed herein with respect to images, fonts, and text fragments often apply to speech samples, user speech models, and phonemes.

9.1. Appropriate Device Optimization Scanning devices for use with the described systems are often small, portable and low power. Scanning devices can only capture a few words at a time, and some implementations do not capture the entire character at once, but instead capture a horizontal slice through the text and put together many such slices to infer the text Form a recognizable signal that can be recognized. The scanning device can also have very limited processing power or storage, but in some embodiments, all of the OCR process itself can be performed, and many embodiments may be later in some cases. Rely on the connection to a more powerful device to convert the captured signal into text. Finally, the scanning device may have very limited capabilities for user interaction, so any request for user input will be postponed later or "best" to a higher extent than is currently common. It may be necessary to operate in "estimate" mode.

9.2. "Uncertain" OCR
The main new feature of the OCR in the described system is to examine images of text that are generally present elsewhere and can be retrieved in digital form. Therefore, accurate transcription of the text is not always required from the OCR engine. The OCR system can output a set or matrix of possible matches, possibly including probability weights, which can be used to search for digital originals.

9.3. Iterative OCR-estimation, disambiguation, estimation ...
If the device performing recognition can contact the document index at the time of processing, the OCR process can be notified by the content of the document corpus as it progresses, and in some cases can provide significantly better recognition accuracy. .

  Such a connection also allows the device to notify the user when enough text has been captured to identify the digital source.

9.4. Use of possible rendering knowledge If the system has knowledge of the possible print rendering aspects of the document, such as the font typeface used for printing, or the layout of the page, or which sections are italicized This can help the recognition process (section 4.1.1).

9.5. Font caching—Determine font on host and download to client Once candidate source text in the document corpus is identified, the font or rendering can be downloaded to the device to aid recognition.

9.6. Autocorrelation and character offset The component characters of a text fragment can be the most appreciated method of representing a fragment of text that can be used as a document signature, but attempt to locate the text fragment in a digital document and / or database In other words, or in disambiguating the display of the text fragment into a readable form, other displays of text that do not need to use the actual text of the text fragment can work well enough. Other displays of text fragments can provide the advantage that there is no display of actual text. For example, optical character recognition of text fragments differs from other representations of captured text fragments that can be used to search and / or recreate text fragments without using optical character recognition for the entire fragment. In this case, an error is likely to occur. Such a method may be more suitable for some devices for use with current systems.

  Those skilled in the art and others will appreciate that there are many ways to describe the appearance of text fragments. Such characterizations of text fragments can include, but are not limited to, word length, relative word length, character height, character width, character shape, character frequency, token frequency, and the like. In some embodiments, the offset between matching text tokens (ie, the number of intervening talks plus one) is used to characterize text fragments.

  Conventional OCR uses knowledge of fonts, character structures, and shapes to attempt to determine characters in scanned text. Embodiments of the present invention are different, that is, use various methods that use the rendered text itself to assist the recognition process. These embodiments use characters (or tokens) to “recognize each other”. One way to refer to such self-recognition is “template matching”, which is similar to “convolution”. In order to perform such self-recognition, the system slides a copy of the text horizontally on the system itself to check the matching area of the text image. Conventional template matching and convolution techniques include a variety of related techniques. These techniques for tokenizing and / or recognizing characters / tokens are collectively referred to herein as “self” when text is used to correlate with self-component parts in matching characters / tokens. This is called “correlation”.

  At the time of autocorrelation, the matched completely connected area is targeted. This happens when a character (or group of characters) overlaps another instance of the same character (or group). The matching fully connected region automatically provides text tokenization into component tokens. As the two copies of the text slide past each other, an area where an exact match has occurred (ie, all the pixels in the vertical slice match) is identified. If the characters / tokens self-match, the horizontal range of this match (eg, the connected matched portion of the text) also matches.

  At this stage, there is no need to determine the actual identity of each token (ie, the specific letter, number or symbol corresponding to the token image, or a group thereof), and an offset to the next occurrence of the same token in the scanned text Please note that it is only necessary. The offset number is an interval (number of tokens) until the next occurrence of the same token. If the token is unique within the text string, the offset is zero (0). The sequence of token offsets thus generated is a signature that can be used to identify the scanned text.

  In some embodiments, the token offset determined for the scanned token string is compared to an index that indexes the corpus of the electronic document based on the token offsets of these contents (section 4.1). .2). In another embodiment, the token offset determined for the scanned token string is converted to text and compared to a conventional index that indexes the corpus of the electronic document based on these contents.

  As described above, if the capture process consists of spoken speech samples, a similar token correlation process can be applied to the speech fragments.

9.7. Font / character "self-recognition"
Conventional template matching OCR compares the scanned image with a library of character images. In essence, the alphabet is stored with each font, and the newly scanned image is compared with this stored image to find a matching character. The process generally has an initial delay until the correct font is identified. Since most documents use the same font throughout, the subsequent OCR process is relatively quick. Thus, subsequent images can be converted to text by comparison with a recently identified font library.

  The shape of the character of the most commonly used font is relevant. For example, in most fonts, the letter “c” and the letter “e” are visually related, such as “t” and “f”. The OCR process is extended by taking advantage of this relationship to construct a template for characters that have not yet been scanned. For example, if a reader scans a short string of text from a paper document of a font that has not been encountered before, the system does not have a set of image templates that compare the scanned images, the system Even if you have never encountered all the characters, you can take advantage of certain estimated relationships between characters to construct a font template library. The system can then use the configured font template library to recognize subsequent scan text and further improve the configured font library.

9.8. Sending unrecognized servers (including graphics) If the image cannot be mechanically transferred to a form suitable for use in the search process, for manual transfer that can be performed by the user for later use, or The image itself can be saved for later processing when different resources become available to the system.

10. P-Commerce Many of the operations that can be performed by this system will result in some commercial transaction. The phrase “P-commerce” is used herein to describe commercial activities initiated from paper through the system.

10.1. Selling a document from a physical printed copy When a user captures text from a document, the document can be provided to the user for purchase in either paper or electronic form. Also, relevant documents can be provided to the user, such as those cited or referred to in paper documents, or related to similar objects, or by the same author.

10.2. Sales of other things initiated or facilitated by paper The capture of text can be linked to other commercial activities in various ways. The captured text can be in a catalog that is specifically designed to sell the item, in which case the text will be fairly directly related to the purchase of the item (section 18.2). . The text can also be part of the advertisement, in which case the sale of the advertised item can continue to occur.

  In another case, the user captures other text that can infer potential interest in commerce. For example, readers of a novel set for a particular country may be interested in holidays for that country. People reading new car reviews may be considering buying a new car. The user may capture specific fragments of text that are known to result in some commercial opportunity being presented to the user or may be a side effect of the user's capture activity. it can.

10.3. Capturing labels, icons, serial numbers, barcodes on items that result in a sale There may be actual text or symbols printed on the item or its package. Examples include serial numbers or manufacturing IDs often found on labels on the back or bottom of some electronic devices. The system can provide the user with a conventional way to purchase one or more of the same items by capturing the text. Manuals, support or repair services can also be provided.

10.4. Contextual advertisements In addition to capturing text directly from advertisements, the system allows for a new kind of advertisements based on what people are reading, although not necessarily obvious in rendered documents.

10.4.1. Ads based on scan context and history In traditional paper publications, ads typically spend a lot of space on the text of newspaper articles and can place a limited number of ads around a particular article. . In the system described, advertisements can be associated with individual words or phrases and can be selected according to the particular interest indicated by the user capturing the text and possibly considering the history of past scans. .

  With the described system, purchases can be tied to a particular printed document, and advertisers can get significantly more feedback on the effectiveness of advertising in a particular print publication.

10.4.2. User context and history based advertising The system can collect a large amount of information about other aspects of the user's context for their own use (section 13), and an estimate of the user's geographic location is a good example. is there. Such data can be used to coordinate advertisements presented to users of the system.

10.5. Compensation models The system enables several new models of compensation to advertisers and marketers. Publishers of printed documents containing advertisements can receive some revenue from purchases generated from these documents. This is true whether or not the advertisement exists in its original print format and can be added electronically by either the publisher, advertiser, or some third party, and the source of such advertisement Can be subscribed to by the user.

10.5.1. Analysis of the statistics generated by the popularity-based reward system can indicate the popularity of certain parts of the publication (Section 14.2). For example, a newspaper can indicate the amount of time a reader spends viewing a particular page or article, or the popularity of a particular columnist. In some situations, it may be appropriate for the author or publisher to receive a reward based on the reader's activity rather than a more traditional metric such as a written word or the number of copies distributed. Authors whose work has become authoritatively read on a subject can be considered different in future contracts than authors whose books sell the same number of copies but rarely open ( (See section 7.6).

10.5.2. Popularity-based advertising Decisions about advertising in documents can be based on statistics about readership. The advertising space around the most popular columnists can be sold for an extra charge. Advertisers can be charged or paid for some time after the document is published based on their knowledge of how the document was received.

10.6. Life Library Based Marketing The “Life Library” or scan history described in Sections 6.1 and 16.1 can be a source of extremely valuable information about user interests and habits. In accordance with appropriate consent and privacy issues, such data can inform the user of the provision of goods or services. Even in anonymous form, the collected statistics can be extremely useful.

10.7. Later sales / information (when available)
Advertising and other opportunities for commerce may not be presented to the user immediately upon text capture. For example, the opportunity to purchase a sequel to a novel cannot be made available when the user is reading the novel, but the system can present the opportunity to the user when the sequel is published.

  The user can capture data regarding purchases or other commercial transactions, but can choose not to start / end the transaction when the capture occurs. In some embodiments, ingestion data is stored in the user's life library, and these life library entries can remain “active” (ie, available when the ingestion occurs). The following dialogue is possible as well as: Thus, the user can review the capture at some later time and optionally complete the transaction based on the capture. Since the system can track the time and place where the original capture occurred, it can properly reward all parties involved in the transaction. For example, an author who wrote a story that was displayed adjacent to an advertisement that the user took in data, and a publisher who published the story, visited the life library six months later, and identified it from the history. You may be rewarded if you choose to capture and select “Purchase this item on Amazon” from the pop-up menu (which can be similar or identical to the menu that is optionally presented at the time of capture) Is possible.

11. Operating System and Application Integration Modern operating systems (OS) and other software packages have many features that can be used to advantage in the described system and are better for this use. It can be modified in various ways to provide a platform.

11.1. Incorporating scan and print related information into metadata and indexing New and next file systems and their associated databases often have the ability to store various metadata associated with each file. Conventionally, this metadata includes the ID of the user who created the file, the creation date and time, the last modification, and the last use. Newer file systems can store additional information such as keywords, image features, document sources, user comments, etc., and some systems can arbitrarily extend this metadata. it can. Thus, the file system can be used to store information that is useful for implementing the current system. For example, the date that a given document was last printed may be stored by the file system so that it can detail which text was taken from paper by who and when using the described system. it can.

  Operating systems are also beginning to incorporate search engine functionality that allows users to find local files more easily. These functions can be used advantageously by the system. This means that many of the search related concepts discussed in Sections 3 and 4 apply not only to today's Internet-based and similar search engines, but also to any personal computer.

  In some cases, a particular software application will include support for the system in addition to the functionality provided by the OS.

11.2. OS support for capture devices As the applicability of capture devices extends beyond a single software application, support for mice and printers becomes more common as the use of capture devices such as pen scanners becomes more common. In much the same way as provided, it would be desirable to incorporate capture device support into the operating system. The same is true for other aspects of system operation. Several examples are discussed below. In some embodiments, the entire described system or its core is provided by the OS. In some embodiments, system support is provided by an application programming interface (API) that can be used by other software packages, including those that directly implement aspects of the system.

11.2.1. Support for OCR and other recognition technologies Most methods for capturing text from rendered documents include some recognition software to interpret the source data (usually scanned images or some spoken language) as text suitable for use in the system. Need. In the past, the use of OCR has usually been limited to a narrow range of applications, and some OSs include support for speech or handwriting recognition, but it is not very common for the OS to include OCR support.

  When the recognition components become part of the OS, these recognition components can better utilize other functions provided by the OS. Many systems include, for example, spelling dictionaries, grammar analyzers, internationalization and localization functions, and in particular these can be customized for a particular user to include words and phrases commonly encountered by the user. All of these can be advantageously used for the recognition process by the described system because it can.

  If the operating system includes full-text indexing capabilities, these can be used to notify the recognition process as described in Section 9.3.

11.2.2. Actions Taken During Scanning When an optical scan or other capture occurs and is presented to the OS, the default action can be taken if the other subsystem does not claim capture ownership. Examples of default behavior are presenting alternative choices to the user or submitting the captured text to the OS's built-in search function.

11.2.3. When the OS finds a digital source of a rendered document that has default behavior for a particular document or document type, the OS takes the standard action that will be taken when that particular document or document of that classification is scanned. Can have. Applications and other subsystems can register with the OS as potential handlers for a particular type of capture, similar to notifications by applying these functions that handle certain file types.

  The markup data associated with the rendering document, or capture from the document, can include instructions to the operating system to launch a particular application and pass arguments, parameters, data, etc. to the application.

11.2.4. Interpretation of gestures and mapping to standard behavior Section 12.1.3 discusses the use of “gestures”, particularly in the case of optical scanning, where the specific movement performed by the handheld scanner is Can represent standard actions such as marking the beginning and end of

  This is similar to actions such as pressing the shift key on the keyboard while selecting a text area using the cursor keys, or scrolling the document using the wheel on the mouse. Such actions by the user are sufficiently standard to be interpreted in a system wide manner by the OS, thus ensuring consistent behavior. The same is desirable for scanner gestures and other scanner-related operations.

11.2.5. Setting Responses to Standard (and Non-Standard) Icon / Text Print Menu Items Similarly, items with text or other symbols can cause standard behavior when scanning, and the OS can select these Can be made possible. In one embodiment, the OS retrieves a copy of the document by printing the text “[PRINT]” in some document and prints it. The OS can also provide a way to register such actions and associate them with a particular scan.

11.3. Support in system GUI components for general scan-initiated operation Most software applications are substantially based on standard graphical user interface components provided by the OS.

  Developers can use these components to help ensure consistent behavior across multiple packages, e.g. left in some text editing context without requiring all programmers to implement the same functionality separately. Pressing the cursor key moves the cursor to the left.

  Similar consistency in these components is desirable when operations are initiated by text capture or other aspects of the described system. Some examples are given below.

11.3.1. Interface for finding specific text content A common use of the system is for a user to scan an area of a paper document, as well as open a corresponding electronic document in a software package that the system can view or edit and scan into that package. Scroll to the highlighted text and highlight it (Section 12.2.1). The first part of this process of finding and opening electronic documents is usually provided by the OS and is standard among software packages. However, the second part, which locates a specific part of text in a document and causes a package to scroll to this part and highlight it, is not yet standardized and may be implemented differently for each package. Many. The availability of a standard API for this function can greatly improve the operation of this aspect of the system.

11.3.2. Text interaction Once a portion of text is located in a document, the system may want to perform various operations on that text. As one example, the system can request surrounding text, which can result in the system accessing the entire sentence or paragraph that contains them with the user's capture of several words. Again, this function is not implemented in every part of the text handling software, but can be effectively provided by the OS.

11.3.3. Some of the operations enabled by the context (pop-up) menu system will require user feedback, which can be optimally requested within the context of the application handling the data. In some embodiments, the system uses an application pop-up menu conventionally associated with right clicking a mouse button on some text. The system inserts additional options into such menus and displays them as a result of operations such as scanning a paper document.

11.4. Web / Network Interface In today's increasingly networked world, most of the functionality available on individual machines can also be accessed over the network, with the exception of the functions associated with the described system. Absent. As an example, in an office environment, many paper documents received by a user may have been printed by other users' machines on the same corporate network. In response to the capture, a system on one computer can query other machines for documents that can respond to the capture according to appropriate authorization controls.

11.5. Preservation resulting from document printing An important factor in the integration of paper and digital documents is to maintain as much information as possible about the transition between them. In some embodiments, the OS keeps a simple record of when and when a document was printed by. In some embodiments, the OS takes one or more additional actions that are more suitable for use with the system. Examples include the following.
Save a digitally rendered version of every printed document along with information about the source from which the document is printed.
Save a subset of useful information about the printed version that can aid in future scan interpretation (eg, the font used and where the line breaks are made).
• Save a version of the source document associated with any printed copy.
• Automatically index documents when printing and store the results for future searches.

11.6. My Documents (Printed / Scanned) The OS often maintains a certain category of folders or files of particular importance. The user's document can be found by convention or design, for example in the “My Documents” folder. The standard file opening dialog can automatically include a list of recently opened documents.

  In an OS optimized for use with the described system, such categories can be expanded or augmented to take into account user interaction with the paper version of the storage file. Categories such as “Printed My Documents” or “Recently Read My Documents” can be usefully identified and incorporated into operations.

11.7. OS-Level Markup Hierarchy Important aspects of the system are usually provided using the “markup” concept discussed in Section 5, so that the OS can access multiple applications as well as the OS itself. Clearly, it would be advantageous to have such markup support provided. Furthermore, the markup layer is provided by the OS based on the inherent knowledge of the document under control and the functions that can be provided.

11.8. Use of OS DRM functionality An increasing number of operating systems support some form of “digital rights management”, ie the ability to control the use of specific data according to the rights granted to specific users, software entities or machines It is coming. This can, for example, prohibit unauthorized copying or distribution of certain documents.

12 User Interface The user interface of the system is either completely on the PC when the capture device is relatively dam and connected to the PC by cable, or the capture device is high performance and each is significant. If it has processing power, it can be present entirely on the device. In some cases, some functionality is present in each component. Some or all of the functions of the system can also be implemented on other devices such as mobile phones or PDAs.

  Accordingly, the description in the following sections shows what may be desirable for a particular implementation, but these are not necessarily appropriate for all and can be modified in several ways.

12.1. On the capture device For all capture devices, especially optical scanners, the user's attention is generally on the device and paper during scanning. Thus, it is highly desirable that any input and feedback required as part of the scanning process does not unnecessarily require the user's attention to be elsewhere (eg, on a computer screen).

12.1.1. Scanner Feedback Handheld scanners can have a variety of ways to provide feedback to a user about a particular situation. The most obvious types are those that the scanner incorporates directly into the indicator light or full display, and those that are audible that the scanner can emit beeps, clicks, or other sounds. An important alternative is a high-performance display that allows the scanner to vibrate, emit a buzzer, or tactile feedback that can stimulate the user's touch sensation, and project something from the light color spot onto the paper Including projection feedback.

Important immediate feedback that can be provided on the device includes:
• Feedback in the scanning process-user scans at too fast speed, too large angles, or too high or too low drift on a particular line.
• Sufficient content-well scanned to make sure it finds a match if it exists-important for separation operations.
Known context-the source of the located text.
Known Unique Context-One unique source of located text.
Content availability-an indication of whether content is freely available to users or costly.

  Many of the user interactions normally associated with later stages of the system can also be performed on the capture device, for example if they have sufficient capabilities to display some or all of the document.

12.1.2. Scanner control The device can provide the user with various ways to allow input in addition to basic text capture. Even if the device is closely associated with a host machine that has input options such as a keyboard and mouse, it can cause confusion for the user to switch between operating the scanner and using the mouse, for example.

  The handheld scanner can have buttons, scroll / jog wheels, touch sensitive surfaces, and / or accelerometers to detect device movement. Some of these allow for a richer set of interactions while holding the scanner.

  For example, in response to a scan of a text, the system presents the user with a set of several possible matching documents. The user uses the scroll wheel on the side of the scanner to select one from the list and clicks the button to confirm the selection.

12.1.3. Gestures The main reason for moving the scanner across the paper is to capture text, but some movements can be detected by the device and used to indicate the intentions of other users. Such movement is referred to herein as a “gesture”.

  As one example, a user may indicate a large area of text by scanning the first few words in a conventional left-to-right order and then the reverse order, ie the last few words from right to left. it can. The user can also indicate the vertical range of text of interest by moving the scanner down the page over several lines. A backward scan can indicate the cancellation of a previous scan operation.

12.1.4. Online / Offline Operation Many aspects of the system may rely on network connectivity either between system components such as scanners and the host laptop, or externally in the form of connections to corporate databases and Internet search. it can. This connectivity is not necessarily present, but some or all of the system may be considered “offline”. It is desirable that the system can continue to function effectively in these situations.

  The device can be used to capture text when not in contact with other parts of the system. A very simple device can simply store image or audio data associated with capture, ideally associated with a timestamp indicating the time of capture. The various captures can be uploaded to other parts of the system the next time the device contacts the system and then processed. The device can also upload other data related to capture, such as audio annotations related to optical scans, or location information.

  Also, a higher performance device can perform some or all of the system operations despite being disconnected. Various techniques for improving the functionality to do this are discussed in section 15.3. Often it is possible to perform some but not all of the desired operations while offline. For example, text can be recognized, but source identification may depend on a connection to an Internet-based search engine. Thus, in some embodiments, the device stores sufficient information about how far each operation has progressed relative to the rest of the system and is efficiently advanced when connectivity is restored. .

  The operation of the system will generally benefit from connectivity that is readily available, but may have the benefit of processing multiple batches and then processing as a batch. For example, as discussed in Section 13 below, the identification of the source of a particular capture can be greatly improved by examining other captures made by the user at about the same time. In a fully connected system where raw feedback is provided to the user, the system can only use past captures when processing the current capture. However, if the capture is one of the batches stored by the device when offline, the system can take into account any data available from previous captures as well as subsequent captures when doing its analysis become.

12.2. On-host devices Scanners often communicate with some other device such as a PC, PDA, phone, or digital camera to perform many of the functions of the system, including more detailed interaction with the user.

12.2.1. Operations Performed in Response to Capture The host device can initiate various operations upon receiving capture. The incomplete list of possible activities performed by the system after locating and the corresponding electronic document associated with the capture and location within that document are as follows:
• Details of capture can be stored in the user's history. (Section 6.1)
Documents can be retrieved from local storage or remote locations. (Section 8)
Update operating system metadata and other records related to the document. (Section 11.1)
The markup associated with the document can be examined to determine the next related operation. (Section 5)
Can start editing, viewing, or operating a software application on a sentence. Application selection may depend on the source document, or the content of the scan, or some other aspect of the capture. (Sections 11.2.2, 11.2.3)
The application can scroll to the capture location, highlight the capture location, move the insertion point to the capture location, or display the capture location. (Section 11.3)
The exact range of captured text can be modified to select, for example, the entire word, sentence, or paragraph around the captured text. (Section 11.3.2)
The user can be given the option to copy the captured text to the clipboard or perform other standard operating system or application specific operations on the captured text.
An annotation can be associated with a document or captured text. These annotations can arise from immediate user input or can have been captured early, for example in the case of audio annotations associated with optical scans. (Section 19.4)
The markup can be examined to determine a further set of possible operations for the user to select.

12.2.2. The appropriate action taken by the context pop-up menu system may be obvious, but may require selections made by the user. One good way to do this is through the use of a “pop-up menu” or, if the content is also displayed on the screen, use a so-called “context menu” that appears near the content. (See section 11.3.3). In some embodiments, the scanner device projects a pop-up menu onto the paper document. The user can use a conventional method such as a keyboard and mouse, or use a control mechanism on the capture device (section 12.1.2), gesture (12.1.3), or scanner. By interacting with the display (section 12.2.4), one can select from these menus. In some embodiments, pop-up menus that can appear as a result of capture include default items that represent actions that are performed if the user does not respond, for example, if the user ignores the menu and performs another capture.

12.2.3. Disambiguation feedback When a user begins capturing text, there will initially be several documents or other text locations that may match. When more text is captured and other factors are considered (section 13), the number of candidate locations is reduced until the actual location is identified, and no further user input is required to further remove ambiguity Then it is impossible. In some embodiments, the system provides a real-time display of documents or locations found, for example, in the form of lists, thumbnail images or text segments, so as to reduce the number of elements in the display as capture continues. To do. In some embodiments, the system displays thumbnails of all candidate documents, where the thumbnail size or position depends on the probability of an exact match.

  Once the capture has been uniquely identified, this fact can be highlighted to the user using, for example, audio feedback.

  Captured text exists in many documents and may be recognized as a quote. The system can indicate this on the screen, for example by grouping documents that contain references around the original source document.

12.2.4. Scanning from the screen Some optical scanners can capture text displayed on the screen as well as on paper. Thus, the term rendering document is used herein to indicate that printing on paper is not the only form of rendering, and capturing text or symbols used by the system is when the text is displayed on an electronic display. Used to indicate that it can be of equal value.

  A user of the described system may be required to interact with the computer screen for various other reasons, such as selecting from a list of options. It may be inconvenient for the user to put the scanner down and start using the mouse or keyboard. Other sections have described physical control mechanisms (section 12.1.2) or gestures (section 12.1.3) on the scanner as input methods that do not require any changes to this tool, but the scanner itself appears on the screen itself. Scanning some text or symbols using is an important alternative provided by the system.

  In some embodiments, the scanner's optics allow the scanner to be used in a manner similar to a light pen, and in some cases actually with the help of special hardware or software on the computer. Sense position directly on screen without having to scan text.

13. Context Interpretation An important aspect of the described system is to use other factors besides simply capturing text strings to help identify the document in use. While a moderate amount of text capture can often uniquely identify a document, in many cases it will be a small number of candidate documents. One solution is to prompt the user to confirm the document to be scanned, but a preferred alternative is to use other factors to automatically narrow the possibilities. Such supplemental information can dramatically reduce the amount of text that needs to be captured and / or improve the reliability and speed with which locations in the corresponding electronic document can be identified. This additional material is called “context” and is described in section 4.2.2. Explained briefly. Let's examine it further here.

13.1. System and Capture Context Perhaps the most important example of such information is the user's capture history.

  In particular, if a previous capture was made in the last few minutes, any given capture is likely to originate from the same document as the previous or related documents (Section 6.1). .2). In contrast, if the system detects that the font has changed between two scans, they are likely from different documents.

  The user's longer acquisition history and reading habits are also useful. They can also be used to develop models of user interests and associations.

13.2. User's Real World Context Another example of a useful context is the user's geographic location. The user in Paris is much more likely to read “Le Monde” than “Seattle Times”, for example. Thus, the timing, size, and geographical distribution of the printed version of the document can be important and can be inferred to some extent from system operation.

  For example, for a user who always reads one type of issue during work and reads another at lunchtime or on the train home, the time of day may also be relevant.

13.3. Related Digital Context Users' recent use of electronic documents, including those searched or retrieved by more conventional means, can also be a useful indicator.

In some cases, other factors may be useful, such as on a corporate network.
• Which documents have been printed recently?
• Which documents have been recently modified on the corporate file server?
• Which documents have been recently emailed?

  All of these examples can suggest that the user is likely reading a paper version of these documents. In contrast, if the repository in which the document resides can be confirmed that the document has never been printed or sent to a printable location, it can be safely deleted in any search originating from paper. it can.

13.4. Other Statistics-Global Context Section 14 covers the analysis of data streams resulting from paper-based searches, but the popularity of the document to other readers, the timing of its popularity, and the most frequently scanned documents It should be noted that all statistics for this part are examples of additional factors that can be beneficial in the search process. The system brings Google-type page ranking possibilities to the paper world.

  See Section 4.2.2 for some other implications on the use of context for search engines.

14 The use of the data stream analysis system produces a very valuable data stream as a side result. This stream is a record of when and what the user is reading, and in many cases is a record of what the user thinks is particularly valuable. Such data has not previously been available for paper documents.

  Some techniques for which this data can be useful to the system and system users are described in Section 6.1. This section concentrates on other uses. Of course, for any distribution of data about what people are reading, there are substantial privacy issues to consider, but the problem of protecting data anonymity is known to those skilled in the art.

14.1. Document tracking If the system knows which document is being read by any given user, the system can infer who is reading any given document. . This makes it possible to track documents throughout the organization, for example, when they read it, how widely distributed, how long it took to distribute, and others that have expired copies Now you can analyze who is viewing the current version while still working.

  With more widely distributed published documents, tracking individual copies is more difficult, but readership distribution analysis is still possible.

14.2. Reading ranking-popularity of documents and sub-regions In situations where users capture text or other data of particular interest to them, the system may infer the popularity of some documents and specific sub-regions of these documents. it can. This forms a valuable input to the system itself (section 4.2.2) and an important information source for authors, publishers and advertisers (section 7.6, section 10.5). This data is integrated into the search engine and search index, for example, to help rank search results for queries originating from rendered documents, and / or to help rank traditional queries typed into a web browser Useful if you are.

14.3. Analyzing the user-building a profile Knowledge of what the user is reading allows the system to create a very detailed model of the user's interests and activities. This can be useful on an abstract statistical basis—ie, “35% of users who purchase this newspaper read the author's latest book” —however, with individual users as described below Other interactions can also be possible.

14.3.1. Social Networking One example is to connect one user to others with related interests. These can be people known to the user. The system can ask university professors, "Do you know that your colleagues at XYZ University have just read this paper?" The system can ask the user, "Do you want to link with your neighbor reading Jane Air?" Such links can be the basis for automatically forming book clubs and similar social organizations either in the real world or online.

14.3.2. Marketing section 10.6 already mentioned the concept of providing products and services to individual users based on user interaction with the system. For example, current online book sellers often recommend users based on previous interactions with their book sellers. Such recommendations are more useful when they are based on dialogue with actual books.

14.4. Marketing Based on Other Aspects of Data Streams The system has described some of the ways people can publish documents, advertise through them, and other sales that start with paper (Section 10). Some commercial activities may not have any direct interaction with paper documents, but may be affected by these. For example, the knowledge that people in a community spend more time reading the sports section than the financial section of a newspaper can be of interest to those seeking to establish a health club.

14.5. Types of data that can be captured In addition to the statistics considered, such as who is reading which part of which document, where the document is located, the actual content of the captured text Finding out can be of interest.

  In many cases, the user will not only capture some text but also cause some action. For example, reference materials for the document can be sent to an acquaintance by e-mail. Even if there is no information about the user's identity or e-mail visit, the knowledge that someone believes that the document is worth sending by e-mail is extremely useful.

  In addition to the various methods that have been considered for inferring the value of a particular document or part of text, in some situations the user will clearly indicate the value by assigning a rating to it.

  Finally, if it is known that a particular set of users will form a group, for example, if it is known to be an employee of a particular company, the group's total statistics can be used to The importance of a particular document can be inferred.

15. Device Features and Functions The capture device used with the system only requires a technique for capturing text from a rendered version of the document. As described above (section 1.2), this capture can be accomplished through various methods, including taking a picture of a portion of the document, or typing a few words on the cell phone keypad. it can. This capture can be accomplished using a small handheld optical scanner that can record one or two lines of text at a time, or a voice capture device such as a voice recorder where a user reads text from a document. The device used can be a combination of these, for example an optical scanner that can also record voice annotations, and the capture function is incorporated into some other device such as a mobile phone, PDA, digital camera, or portable music player Can do.

15.1. Input and Output Many of the potentially advantageous additional input and output functions for such devices have been described in Section 12.1. These include buttons, scroll wheels, and touchpads as inputs, and displays, indicator lights, voice and haptic transducers as outputs. Devices may incorporate many of these and may be very few. The capture device may be able to communicate with another device that already has these capabilities using, for example, a wireless link (section 15.6) and may incorporate the capture functionality into such other devices. (Section 15.7).

15.2. Connectivity In some embodiments, the device implements most of the system itself. However, in some embodiments, the device often communicates with a PC or other computing device and with a wider world using communication capabilities.

  These communication functions are in the form of a general-purpose data network such as Ethernet, 802.11, or UWB, or a standard peripheral connection network such as USB, IEEE-1394 (Firewire), Bluetooth (trademark), or infrared. When a wired connection such as Firewire or USB is used, the device can receive power through the same connection. In some situations, the capture device can be viewed as if the connected machine is a conventional peripheral such as a USB storage device.

  Finally, in some situations, a device can be used in conjunction with the device or “docked” with another device as a convenient storage device.

15.3. Caching and other online / offline features Sections 3.5 and 12.1.4 addressed the topic of separation operations. The device can still be useful when the capture device has a limited subset of the full functionality of the system and is not in communication with other parts of the system, but in some cases the available functionality is reduced Will be. At the simplest level, the device can record raw image or audio data that is captured and can process it later. However, for the benefit of the user, if possible, identify whether the captured data is sufficient for the task in progress, whether it can or can be recognized, and the source of the data It can be important to give feedback as to whether it can be or can be identified later. The user will then recognize whether this capture operation is valuable. Even if all of the above are unknown, the raw data can be stored so that at least the user can refer to them later. For example, if the scan cannot be recognized by the OCR process, an image of the scan can be presented to the user.

  To illustrate some of the range of options available, both a somewhat less functional optical scanning device and a much more fully equipped device are described below. Many devices occupy an intermediate point between the two.

15.3.1. SimpleScanner-Low Cost Offline Example A simple scanner has a scan head that can read pixels from a page as it moves along the length of a text line. A simple scanner can detect movement along the page and record the pixels with some information about this movement. Simple scanners also have a clock, which allows each scan to be time stamped. The clock synchronizes with the host device when the simple scanner has connectivity. The clock cannot represent the actual time, but the relative time can be determined from the clock, which allows the host to derive the actual time of the scan, or at worst, the elapsed time between scans. .

  Simple scanners do not have enough processing power to perform any OCR itself, but have some basic knowledge of the relationship between general word length, word spacing, and font size. . A simple scanner is a scan that can be read, whether the head movement on the paper is too fast, too slow, or inaccurate, and a given size for the document being identified It has several basic indicator lights that let the user know when it is determined that a sufficient number of words have been scanned.

  The simple scanner has a USB connector and can be plugged into a USB port on the computer where it will be charged. To the computer, it appears to be a USB storage device with a time-stamped data file recorded, and the rest of the system software takes over from this point.

15.3.2. SuperScanner-High Performance Offline Example A superscanner also depends on connectivity for its full operation, but can help make better decisions about data captured while offline It has a significant amount of on-board storage and processing power.

  As the superscanner moves along the line of text, the captured pixels are grouped together and passed to the OCR engine attempting to recognize the text. Numerous fonts, including those from the publications most read by the user, have this font, so that it has a dictionary that is synchronized with the user's spelling checker dictionary on the user's PC and contains many of the words that the user frequently encounters Has been downloaded to help perform the task. The scanner also stores a list of these standard usage words and phrases that can be combined with a dictionary. Scanners can use frequency statistics to help in the recognition process and inform decisions about when a sufficient amount of text has been captured, and more frequently used phrases are useful as criteria for search queries Is unlikely.

  In addition, a complete index of recent newspaper articles, such as an index of books recently purchased by online users from online book sellers, or scanned by users within the last few months, and periodicals that users read most commonly Things are stored on the device. Finally, the titles of thousands of the most popular publications with data available to the system are stored, which allows the user to scan titles without any other information and from specific works You will have a good idea as to whether the capture can later be retrieved in electronic form.

  During the scanning process, the system informs the user that the captured data is of sufficient quality and of sufficient nature that the electronic copy can be retrieved when connectivity is restored. The system knows that the scan was successful and that the context is recognized by one of the onboard indexes, or the associated publication knows that the data is available to the system. Therefore, it often indicates to the user that later retrieval should be successful.

  The superscanner docks in a cradle connected to the PC's Firewire or USB port, at which point, in addition to uploading the captured data, its various on-board indexes and other databases can be used for recent user actions and Updated based on new issue. The super scanner also has a function of connecting to a wireless public network, a function of communicating with a mobile phone via Bluetooth, and a function of communicating with the public network when these functions are available.

15.4. Optical Scan Features Next, some of the features that may be particularly desirable in an optical scanner device are considered.

15.4.1. Flexible positioning and convenient optics One of the reasons why paper is still popular is because it is easy to use, for example, in a wide range of situations where computers are impractical or inconvenient. Therefore, devices that are intended to capture a significant portion of user interaction with paper should be equally convenient to use. This was not a past scanner, and it was somewhat tricky even with the smallest handheld devices. Handheld devices designed to contact the page must be held at an accurate angle to the paper and moved very carefully along the length of the text being scanned. This is acceptable when scanning business reports at the office desk, but can be impractical when scanning phrases from novels while waiting for a train. Scanners based on camera-type optics that operate at a distance from the paper can be equally useful in some situations.

  Some embodiments of the system utilize a scanner that scans in contact with the paper and transmits the image from the page to the optical sensor device using an image conduit that is a fiber optic bundle instead of a lens. Such a device can be shaped so that it can be held in a natural position, for example, in some embodiments, the portion that contacts the page is wedge-shaped and the user's hand used a highlighter pen. You can move on the page more naturally with the same movement. The conduit can be in direct contact with or in close proximity to the paper and have a replaceable transparent tip that can protect the image conduit from possible damage. As described in Section 12.2.4, a scanner can be used to scan from the screen as well as paper, and the tip material can be selected to reduce the possibility of damage to such a display.

  Finally, some embodiments of the device provide light, sound, or haptic feedback if the user scans too early, too late, is not uniform, or the scanned row is drifting too high or too low. Use will provide feedback to the user during the scanning process.

15.5. Security, Identity, Authentication, Personalization, and Billing As described in Section 6, capture devices can form an important part of identification and authorization for secure transactions, purchases, and various other operations. . Thus, in addition to the circuitry and software necessary for such a role, the capture device can be made more secure, such as a smart card reader, RFID, or keypad for typing a PIN. Can be incorporated.

  The capture device can include various biometric sensors to help identify the user. For example, in the case of an optical scanner, the scan head can read a fingerprint. In the case of a voice recorder, the user's voiceprint can be used.

15.6. Device Association In some embodiments, a device can form an association with itself or other neighboring devices to improve its functionality. In some embodiments, for example, the device uses a nearby PC or telephone display to provide more detailed feedback about its operation or use its network connectivity. On the other hand, devices can act as security and identification devices to authenticate operations performed by other devices. Alternatively, the device can simply form an association to function as a peripheral to the device.

  An interesting aspect of such association is that the device capture function can be used to initiate and authenticate the association. For example, a user who wants to identify himself / herself securely to a public computer terminal scans a code or symbol displayed in a specific area of the terminal screen using the scan function of the device and performs key transfer be able to. A similar process can be performed using an audio signal picked up by an audio recording device.

15.7. Integration with other devices In some embodiments, the functionality of the capture device is integrated with other devices already in use. The integrated device can enable sharing of power, data capture and storage functions, and network interfaces. Such integration can be done for convenience only to reduce costs or to allow functions that are not otherwise available.

Some examples of devices that can integrate capture functions are listed below.
Existing peripherals such as a mouse, stylus, USB “webcam” camera, Bluetooth ™ headset, or remote control device:
Another processing / storage device such as a PDA, MP3 player, voice recorder, digital camera, or mobile phone:
Other items that are often carried for convenience, such as watches, jewelry, pens, and car keychains:

15.7.1. Cellular Phone Integration As one example of the benefits of integration, consider the use of a modified cell phone as a capture device.

  In some embodiments, the phone hardware can handle text capture by the phone itself or by the system at the other end of the call if the text capture can be done properly via speech recognition. Is not modified to support the system, such as if it can be stored in the phone's memory for future processing. Many modern phones have the ability to download software that can implement several parts of the system. Such speech capture is likely to be suboptimal in many situations, but accurate speech recognition is a difficult task even at best, for example, when there is significant background noise. The voice function may be optimally used to capture voice annotations.

  In some embodiments, cameras built into many cell phones are used to capture images of text. The phone display, usually functioning as a camera viewfinder, is a representation of text for image quality and OCR suitability, which segments of text are captured, and if OCR can be performed on the phone. Information about the format can be overlaid on the live camera image.

  In some embodiments, the phone is modified to add a dedicated capture function or to provide such functionality in a clip-on adapter or a separate Bluetooth connected peripheral that communicates with the phone. Regardless of the nature of the capture mechanism, integration with modern cell phones has many other advantages. The phone has connectivity to the wider world, which allows queries to be submitted to remote search engines or other parts of the system, and copies of documents to be retrieved for immediate storage or display Means. A telephone typically has enough processing power to perform many of the functions of the system locally and enough storage to capture a reasonable amount of data. The amount of storage is also often expandable by the user. Telephones have reasonably good display and audio capabilities to provide feedback to the user, and often vibration capabilities for haptic feedback. The telephone also has a good power source.

  The most important of all is that the phone is a device that many users already carry.

Part 3-Exemplary Applications of the System This section lists exemplary uses of the system and applications incorporated into the system. This list is purely illustrative and is not meant to be exhaustive.

16. Personal application 16.1. Life Library The Life Library (see also Section 6.1.1) is a digital archive of any important document that a subscriber wants to store and is an embodiment of the system's service set. All important books, magazine articles, newspaper clippings, etc. can be stored in digital form in the Life Library. In addition, subscriber annotations, comments, and notes can be saved with the document. The life library can be accessed via the Internet and the World Wide Web.

  The system creates and manages a life library document archive for subscribers. The subscriber wants the subscriber to store which document in the subscriber's life library by scanning information from the document or by indicating to the system that a particular document is added to the subscriber's life library Indicate. The scan information is typically text from the document, but can also be a barcode or other code that identifies the document. The system accepts the code and uses it to identify the source document. After the document is identified, the system can store a copy of the document in the user's life library or a link to a source from which the document can be obtained.

  In one embodiment of the life library system, it can be checked whether the subscriber is authorized to obtain an electronic copy. For example, if a reader scans text or identifiers from a copy of a New York Times (NYT) article, thereby allowing the article to be added to the reader's life library, the life library system will allow the reader to join the online version of NYT. If NYT verifies that you are subscribed and if you are subscribed, the reader will get a copy of the article stored in your life library account, and if not subscribed, the information identifying the document and the document's The ordering method is stored in the reader's life library account.

  In some embodiments, the system maintains a subscriber profile for each subscriber that includes access rights information. Document access information can be compiled in several ways, two of which are: 1) the subscriber supplies the document access information to the life library system along with the subscriber's account name and password, etc. Or 2) The life library service provider queries the publisher for subscriber information and gives the publisher access to an electronic copy of the document if the life library subscriber is authorized to access the material. Is to respond. If the life library subscriber is not authorized to have an electronic copy of the document, the issuer presents the price to the life library service provider and then provides the customer the option to purchase the electronic document. In this case, the life library service provider pays the issuer directly and later charges the life library customer, or the life library service provider immediately charges the customer's credit card for the purchase. Life Library service providers will receive a certain percentage of the purchase price or a small fixed fee to facilitate the transaction.

  The system can archive documents in the subscriber's personal library and / or some other library for which the subscriber has archiving rights. For example, when a user scans text from a printed document, the life library system can identify the rendered document and its corresponding electronic document. After the source document is identified, the life library system can record information about the source document in the user's personal and group libraries for which the subscriber has archiving rights. A group library is a collaborative archive such as a document repository for groups working together on a project, groups of academic researchers, group weblogs, and others.

  A life library can be in many ways: time series, topic, subscriber level of interest, type of publication (newspaper, book, magazine, technical paper, etc.), reading location, reading time, ISBN, or Dewey. It can be organized in various ways, such as every decimal. In one alternative, the system can learn the classification based on how other subscribers have classified the same document. The system can suggest a classification to the user or can automatically classify the document for the user.

  In various embodiments, the annotation can be inserted directly into the document or can be maintained in a separate file. For example, when a subscriber scans text from a newspaper article, the article is archived in the subscriber's life library with the scanned text highlighted. Alternatively, the article is archived in the subscriber's life library along with the associated annotation file (thus the archived document remains unmodified). Embodiments of the system maintain one copy of the source document in each subscriber's library, one copy in a master library accessible to many subscribers, or a link to a copy maintained by the publisher Can do.

  In some embodiments, the life library stores only user modifications (eg, highlighting) to the document and links to online versions of the document (stored elsewhere). The system or subscriber merges this change into the document when the subscriber next retrieves the document.

  If the annotations are kept in separate files, the source document and the annotation file are provided to the subscriber and the subscriber combines them to create a modified document. Alternatively, the system combines the two files before presenting them to the subscriber. In another alternative, the annotation file is an overlay to the document file and can be overlaid on the document by software in the subscriber's computer.

  Subscribers to the Life Library service pay a monthly fee and let the system maintain the subscriber's archive. Alternatively, the subscriber pays a small amount (eg, micropayment) for each document stored in the archive. Alternatively, the subscriber accesses the subscriber's archive for a fee per access. Alternatively, the subscriber compiles the library and allows others to access the material / annotations on the revenue share model with the life library service provider and copyright owner. Alternatively, the life library service provider receives payment from the publisher when a life library subscriber orders a document (revenue share model with publisher, where the life library service provider is responsible for the share of the publisher's revenue). Receive).

  In some embodiments, the life library service provider acts as an intermediary between the subscriber and the copyright owner (or an agent of a copyright owner such as the Copyright Clearance Center known as CCC). In fact, it facilitates billing and payment for copyrighted materials. The life library service provider uses the subscriber's billing information and other user account information to provide this intermediary service. Basically, a life library service provider can leverage existing relationships with subscribers to purchase copyrighted material on behalf of subscribers.

  In some embodiments, the life library system can store excerpts from documents. For example, when a subscriber scans text from a paper document, the entire document is not archived in the life library, but the area around the scanned text is extracted and placed in the life library. This is particularly advantageous when the document is long, as it preserves the original scan status so that the subscriber does not reread the document to find the part of interest. Of course, a hyperlink to the entire electronic document corresponding to the paper document can be included with the excerpt material.

  In some embodiments, the system may also include an author, publication title, publication date, publisher, copyright owner (or copyright owner's license agent), ISBN, link to the public annotation of the document, reading ranking. Information about documents such as is stored in the life library. Some of this additional information about the document is in the form of paper document metadata. Third parties can create public annotation files for access by people other than themselves, such as the general public. Links to third-party comments about the document are advantageous because reading the other user's annotation file deepens the subscriber's understanding of the document.

  In some embodiments, the system archives material by class. This feature allows life library subscribers to quickly store corresponding electronic documents for the entire class of paper documents without accessing each paper document. For example, if a subscriber scans some text from a copy of National Geographic magazine, the system provides the subscriber with an option to archive all National Geographic back numbers. If the subscriber chooses to archive all back numbers, the life library service provider checks with the National Geographic Society to see if the subscriber is authorized to archive. If not, the life library service provider can mediate the purchase of the right to archive the National Geographic collection.

16.2. Life Saver A variation or extension of the life library concept is a “life saver”, where the system further infers other actions of the user using text captured by the user. By scanning menus from specific restaurants, programs from specific theater performances, timetables at specific railway stations, or articles from local newspapers, the system can infer about user locations and social activities For example, an automatic diary of social activities can be configured like a website. The user can edit and modify the diary, add additional material such as photos, and of course, view the scanned items again.

17. Academic applications Portable scanners supported by the described system have many potential uses in academic settings. Portable scanners can improve student / teacher interaction and enhance the learning experience. Among other uses, students can annotate learning materials to meet student specific needs, teachers can monitor academic performance, and teachers are cited in student assignments. Source material can be verified automatically.

17.1. Children's Books Children's interactions with paper documents such as books are monitored by a reading and writing ability acquisition system that uses a specific set of embodiments of the system. The child uses a portable scanner that communicates with other elements of the literacy learning system. In addition to the portable scanner, the read / write learning system includes a computer having a display and speakers and a database accessible to the computer. The scanner is coupled to a computer (hardwired, short range RF, etc.). When the child looks up an unknown word in the book, the child scans the word with a scanner. In one embodiment, the literacy learning system compares the scanned text with resources in its database to identify the word. The database includes dictionaries, thesaurus, and / or multimedia files (eg, audio, graphics, etc.). After the word is identified, the system reads the word and definition aloud to the child using a computer speaker. In another embodiment, the word and its definition are displayed on a computer monitor by a literacy learning system. Multimedia files for scanned words can also be played through a computer monitor and speakers. For example, if a child reading “Goldilocks and the Three Bears” scans the word “bear”, the system reads the word “bear” and plays a short video about the bear on the computer monitor. be able to. In this way, the child learns the pronunciation of the written word and visually learns the meaning of the word through multimedia expressions.

  A literacy acquisition system provides immediate audio and / or visual information that improves the learning process. The child uses this supplemental information to quickly and deeply understand the written material. The system can be used to teach novice readers to read, to help children acquire more vocabulary, and so on. The system provides the child with information about words that the child is not familiar with or information about words that the child wants more information.

17.2. Read / Write Capability In some embodiments, the system compiles a personal dictionary. When a reader looks up a new, interesting, or particularly useful or problematic word, the reader saves this word (with its definition) in a computer file. This computer file becomes the reader's personalized dictionary. Since this dictionary is generally smaller in size than a normal dictionary, it can be downloaded to a mobile station or associated device, and thus can be made available even if the system is not immediately accessible. In some embodiments, the personal dictionary entry includes an audio file to assist in proper word pronunciation and information identifying the paper document from which the word was scanned.

  In some embodiments, the system creates customized spelling and vocabulary tests for students. For example, when a student reads an assignment, the student can scan an unknown word with a portable scanner. The system stores a list of all words scanned by the student. Later, the system performs a spelling / vocabulary test customized to the student on the associated monitor (or prints the test on the associated printer).

17.3. Music Education The arrangement of notes on the staff is similar to the arrangement of characters in one line of text. The system can capture notes using the same scanning device described above for capturing text in this system, and a similar process that constitutes a search against a database of known musical compositions makes it possible to identify the captured works Can then be the basis for retrieval, playback, or some further action.

17.4. Theft detection The teacher can use the system to detect plagiarism or verify the source by scanning the text from the student's paper document and submitting the scanned text to the system. For example, a teacher who wants to verify that a citation in a student's paper document is from a student-listed source, scans the citation, and the document title identified by the system and the title of the document listed by the student Can be compared. Similarly, the system can use a scan of text from an assignment submitted as the student's original work to determine if the text was copied instead.

17.5. Enhanced textbooks In some embodiments, capturing text from academic textbooks can help students or staff to provide more detailed explanations, more exercises, student-staff discussions on materials, and related past exam questions. Links to examples, further reading on the subject, lecture recording on the subject, etc. (see Section 7.1).

17.6. Language Learning In some embodiments, the system is used to teach a foreign language. For example, when a Spanish word is scanned, the word is read aloud in Spanish with its definition in English.

  The system immediately provides auditory and / or visual information to enhance the new language acquisition process. Readers use this supplemental information to help them understand the material quickly and deeply. The system can be used to teach novice students how to read foreign languages, help students acquire more vocabulary, and so on. The system provides information about words in foreign languages that the reader is not familiar with, or about words that the reader wants more information.

  The interaction between a paper document, such as a newspaper or book, and a reader is monitored by a language skill system. The reader has a portable scanner that communicates with the language skill system. In some embodiments, the language skill system includes a computer having a display and speakers and a database accessible to the computer. The scanner communicates with the computer (hardwired, short range RF, etc.). When a reader looks up an unknown word in an article, the reader scans this word with a scanner. The database includes foreign language dictionaries, thesaurus, and / or multimedia files (speech, graphics, etc.). In one embodiment, the system compares the scanned text with resources in its database and identifies the scanned word. After the word is identified, the system reads the word and its definition to the reader using a computer speaker. In some embodiments, both the word and its definition are displayed on a computer monitor. Multimedia files for grammatical hints associated with scanned words can also be played through a computer monitor and speakers. For example, if the word “to spike” is scanned, the system will pronounce the word “hablar”, play a short audio clip showing the appropriate Spanish pronunciation, and complete the various uses of “hablar”. A list can be displayed. In this way, students learn the pronunciation of the written word, learn the spelling of the word visually through multimedia expressions, and learn how to use verbs. The system can also provide grammatical hints about proper usage of “hablar” along with idioms.

  In some embodiments, the user scans a word or short phrase from the rendered document in a language other than the user's native language (or other language that the user knows reasonably). In some embodiments, the system maintains a priority list of the user's “preferred” languages. The system identifies the corresponding electronic document of the rendered document and determines the location of the scan within the document. The system also identifies a second corresponding electronic document of the document that has been translated into one of the user's preferred languages and determines a location in the translated document that corresponds to the location of the scan in the original document. If the corresponding location is not accurately known, the system identifies a small region (eg, paragraph) that includes the corresponding location of the scanned location. The corresponding translation location is then presented to the user. This provides the user with an accurate translation of a particular usage at the scanned location, including any slang or other idiomatic usage that is often difficult to translate accurately on a word order basis.

17.7. Collection of research material Users who are investigating a particular topic may encounter all types of material, both in print and on the screen, and the user is considered to be related to that topic in a personal archive. You may want to record. The system can automatically perform this process as a result of scanning a short phrase in any part of the material, and can create a suitable reference for publication in a publication on this subject. .

18. Commercial Applications Clearly, commercial activities can be explained from almost any process described herein, but here we will focus on some obvious revenue streams.

18.1. Fee-based search and indexing Conventional Internet search engines typically provide free search for electronic documents and do not charge content providers for including these content in the index. In some embodiments, the system allows users to be billed and / or paid to search engines and / or content providers for the operation and use of the system.

  In some embodiments, subscribers to the services of the system pay a fee for searches that result from scanning paper documents. For example, a stock broker may have read a Wall Street Journal article about a new product offered by Company X. By scanning the company X name from a paper document and agreeing to pay the necessary fee, the stockbroker uses the system to search a special or dedicated database, such as an analyst report Get premium information about the company. The system also indexes documents that are most likely to be read in paper form, for example by indexing all newspapers published on a particular day and ensuring that the newspaper is available before it enters the city. Arrange to prioritize creation.

  Content providers can pay a fee associated with some terms in a search query submitted from a paper document. For example, in one embodiment, the system selects the most preferred content provider based on the additional context for the provider (the context is in this case the content provider paid a fee so that the results list is advanced) That's what it means). In essence, search providers adjust paper document search results based on existing payment agreements with content providers. See also the description of keywords and key phrases in section 5.2.

  If access to specific content needs to be restricted to a specific group of people (such as clients or employees), such content is protected by a firewall and is therefore generally not indexed by a third party. I can't do it. Nevertheless, content providers may wish to provide an index for protected content. In such a case, the content provider can pay the service provider and provide the content provider's index to the system subscriber. For example, a law firm can create an index on all of a client's documents. The document is stored behind the law firm's firewall. However, since the law firm wants its employees and clients to access the document via a portable scanner, it provides the service provider with an index (or pointer to the index) to allow the law firm employee or Search a law firm index when a client submits a search term that scans paper through their portable scanner. The law firm can provide a list of employees and / or clients to the service provider's system to enable this feature, or the system can support the law firm prior to searching the law firm's index. The access right can be verified by inquiring. Note that in the above example, the index provided by the law firm is only that client's document, not the index of all documents in the law firm. Thus, the service provider is only allowed the law firm's client access to the documents that the law firm has indexed for the client.

  There are at least two separate revenue streams that can be the result of a search derived from paper documents, one is the revenue stream from the search function and the other is the revenue stream from the content distribution function. It is. Search function revenue can be generated from paid subscription fees from scanner users, but can also be generated at a charge per search. Content distribution revenue can be shared with content providers or copyright owners (service providers can take a percentage of sales such as small payments for each distribution or a fixed fee) Regardless of whether or not to broker a transaction, it can also be generated by a “query” model where a subscriber orders from an online catalog and takes a fee or a percentage for all items delivered or offered by the system. In some embodiments, the system service provider earns revenue for all purchases made by the subscriber from the content provider during a predetermined time period or some time thereafter when the identified product purchase is made. receive.

18.2. Catalog Consumers can shop from a paper catalog using a portable scanner. The subscriber scans information from the catalog to identify the catalog. This information is text from the catalog, a barcode, or another identifier for the catalog. The subscriber scans for information identifying the product that they want to purchase. The catalog address label can include a customer identification number that identifies the customer to the catalog vendor. In this case, the subscriber can also scan this customer identification number. The system acts as an intermediary between the subscriber and the vendor, facilitating catalog purchases by providing the vendor with a customer selection and customer identification number.

18.3. Coupons Consumers scan paper coupons and store an electronic copy of the coupon in a scanner or remote device such as a computer for later retrieval and use. The advantage of an electronic storage device is that the consumer is freed from the burden of carrying paper coupons. A further advantage is that electronic coupons can be taken from any location. In some embodiments, the system can track coupon expiration dates, notify consumers about coupons that are about to expire, and / or delete expired coupons from storage. An advantage for the coupon issuer is that they may receive more feedback regarding who is trying to use the coupon and when and where the coupon was captured and used. .

19. General application 19.1. Format The system can be used to automatically populate electronic documents that correspond to paper formats. The user scans some text or barcode that uniquely identifies the paper form. The scanner communicates the form identity and information identifying the user to a nearby computer. Nearby computers have an internet connection. A nearby computer can access a first database of forms and a second database (such as a service provider's subscriber information database) that contains information about the user of the scanner. A nearby computer accesses the electronic version of the paper form from the first database and automatically populates the fields of the form from the user information obtained from the second database. The nearby computer then emails the completed form to the designated recipient. Alternatively, the computer can print the completed form on a nearby printer.

  Rather than accessing an external database, in some embodiments, the system has a portable scanner that contains the user's information, such as in an identity module, SIM, or security card. The scanner provides information identifying the form to a nearby PC. A nearby PC accesses the electronic form, queries the scanner for any necessary information, and fills in the form.

19.2. The business card system can be used to automatically populate electronic address books or other contact lists from paper documents. For example, when a business card of a new acquaintance is received, the user can capture an image of the business card using the user's mobile phone. The system can locate the electronic copy of the business card and use it to update the mobile phone's onboard address book with the contact information of the new acquaintance. Electronic copies can contain more information about new acquaintances than stuffing business cards. In addition, the onboard address book can also store a link to the electronic copy, so that any changes to the electronic copy are automatically updated in the mobile phone address book. In this embodiment, the business card optionally includes a symbol or text indicating the presence of an electronic copy. In the absence of an electronic copy, the mobile phone can use the OCR and standard business card format knowledge to fill in an address book entry for a new acquaintance. Symbols can assist in the process of extracting information directly from an image. For example, a telephone icon next to a business card telephone number can be recognized to specify the location of the telephone number.

19.3. Proofreading / editing The system can extend the proofreading and editing process. One way that the system can extend the editing process is based on linking the editor's interaction with a paper document to its corresponding electronic document. As an editor reads a paper document and scans various parts of the document, the system will make appropriate annotations or edits to the corresponding electronic document of the paper document. For example, if an editor scans a portion of text and uses a scanner to perform a “new paragraph” control gesture, the computer communicating with the scanner will be “new” at the location of the scanned text in the electronic copy of the document. Insert a paragraph break.

19.4. Voice Annotation A user can annotate a document by scanning a portion of the text from the document and then making a voice recording associated with the scanned text. In some embodiments, the scanner has a microphone for recording the user's verbal annotations. After the verbal annotation is recorded, the system identifies the document from which the text was scanned, locates the scanned text within the document, and attaches a voice annotation at that point. In some embodiments, the system converts speech to text and attaches annotations as text comments.

  In some embodiments, the system keeps the annotation separate from the document and only a reference to the annotation is kept with the document. The annotation then becomes an annotation markup layer for the document for a particular subscriber or group of users.

  In some embodiments, for each capture and associated annotation, the system identifies the document, opens the document using a software package, scrolls to the location of the scan, and plays the audio annotation. The user can then interact with the document with reference to voice annotations, proposed changes, or other comments recorded by himself or someone else.

19.5. Text Help Using the described system, a paper document can be extended with an electronic help menu. In some embodiments, the markup layer associated with a paper document includes help menu information about the document. For example, when a user scans text from a particular part of a document, the system checks the markup associated with the document and presents a help menu to the user. The help menu is presented on the scanner's display or on an associated nearby display.

19.6. Depending on the situation of use with the display, it may be advantageous to be able to scan information from a television, computer monitor, or other similar display. In some embodiments, a portable scanner is used to scan information from computer monitors and televisions. In some embodiments, the portable optical scanner has an illumination sensor and is optimized to work with conventional cathode ray tube (CRT) display technologies such as rasterization, screen blanking, etc.

  An audio capture device that operates by capturing the voice of a user who is reading text from a document typically operates whether the document is on paper, on a display, or on some other medium. Become.

19.6.1. Public booth and dynamic session ID
One application of direct scanning of the display is the device association described in section 15.6. For example, in some embodiments, a public booth displays a dynamic session ID on its monitor. The booth is connected to a communication network such as the Internet or a corporate intranet. The session ID changes periodically, but changes whenever a new session ID is displayed to all users, at least using the booth. To use the booth, the subscriber scans the session ID displayed in the booth, and by scanning the session ID, the user delivers content resulting from scanning the printed document or from the booth screen itself In order to do so, the system is informed that the booth is temporarily associated with the user's scanner. The scanner can communicate directly to the system the session ID and other information (such as a serial number, account number, or other identification information) that authenticates the scanner. For example, the scanner can communicate directly with the system by sending a session initiation message via the user's mobile phone (paired with the user's scanner via Bluetooth ™) (where "Direct" means you don't pass a message through the booth). Alternatively, the scanner establishes a wireless link with the booth and uses the booth's communication link by transferring session initiation information to the booth (possibly via a short range RF such as Bluetooth ™). In response, the booth sends session initiation information to the system over the Internet connection.

  The system can prevent others from using a device that is already associated with the scanner during the period (or session) in which the device is associated with the scanner. This feature is useful to prevent others from using the public booth before the end of another person's session. As an example of this concept for using a computer in an Internet cafe, the user scans a bar code on the PC monitor that the user wants to use, and in response, the system sends a session ID to the display monitor. The user starts the session by scanning the session ID from the monitor (or entering the session ID via the keypad or touch screen or microphone on the portable scanner), and the system stores the session ID in its database. Because it is associated with your scanner serial number (or other identifier that uniquely identifies the user's scanner), another scanner will scan the session ID during the user's session and will not be able to use the monitor. The scanner communicates with the PC associated with the monitor (via a wireless link such as Bluetooth ™, a hard wire drink such as a docking station, etc.) or with the system via another means such as a mobile phone. Communicate directly (ie, without going through a PC).

Part 4-System Details To trigger an action such as an advertisement in response to capturing a keyword optically or acoustically from a rendered document or in response to identifying a document based on the captured keyword The software and / or hardware system will be described (“system”). In some cases, the system presents advertisements for keywords, displays annotations, or modifies or applies actions. As used herein, a keyword means one or more words, icons, symbols, or images. Although the terms “word” and “multiple words” are often used in this application, in some embodiments, icons, symbols, or images may be used. A keyword as used herein also refers to a phrase composed of one or more adjacent symbols. As used herein, keywords include words associated with topics or subjects that are identified in response to capture and considered with the rendered document or portion of the rendered document. The keywords can optionally include a class of objects that can be recognized by regular expression algorithms or image processing. Such classes of objects can include email addresses, mailing addresses, telephone numbers, URLs, hyperlinks, and other pointers to content, quotes, trademarks, logos, full names, times, dates, etc. .

  Keywords are considered "overloaded", i.e., keywords have some associated meaning or action beyond the general (e.g. visual) meaning for the user, such as text or symbols. In some embodiments, the association between keywords and meanings or actions is set by a markup process or data. In some embodiments, the association between a keyword or document and a meaning or action is known to the system when it is captured or identified. In some embodiments, the association between keywords or documents and meanings or actions is established after capture or identification has taken place.

  In some embodiments, the system uses the content of the document to identify the document and to trigger and select advertisements presented to the user. In some cases, the system can parse the document and associate the content of the document with one or more keywords. In some cases, the system selects an advertisement (action) based on the content of the entire document. In some cases, the system selects an advertisement based on a portion of the document that includes or is near the captured text. In some cases, the system selects an advertisement based on the content of the document that is not used when identifying the document.

  In some embodiments of the described system, interaction with a keyword in a rendered document includes that the capture from the document specifically includes the keyword, or that the keyword associated with the identified document is a specific keyword. You don't need to be. Ingestion includes the keyword as a whole, overlaps with the keyword (includes part of the keyword), is close to the keyword (eg, on the same paragraph or on the same page), or is similar to information contained in the keyword Incorporating information or related information (eg, words, icons, tokens, symbols, images) can trigger an action associated with the keyword. The action associated with the keyword can be invoked when the user captures a synonym for the word contained in the keyword or when the document is associated with a synonym for the keyword. For example, if the keyword includes the word “cat” and the user captures text including the word “feline”, an action associated with “cat” can optionally be invoked. Alternatively, if the user captures somewhere on the page that includes the word “cat” or the word “feline”, optionally an action associated with a keyword that includes “cat” can be invoked.

  Similarly, if the system identifies a document that contains “Feline”, parses the contents of the document, and further determines a keyword for the document, the system will perform an action (advertisement message) associated with the keyword “cat”. Etc.) can be called.

  In some embodiments, specific instructions and / or data specifying how captures are related to keywords and what specific actions result from these captures are stored as markup in the system. The

  In some embodiments, the action taken in conjunction with the keyword is determined in part by how the capture was performed. The proximity of keywords, overlaps to keywords, inclusions in addition to keywords, and inclusions that strictly contain keywords can each result in a different set of actions. Incorporating the keyword “IBM” with no material around it allows the user's browser to be directed to the IBM website. Capturing an IBM within the surrounding sentence can cause an IBM advertisement to be displayed while the system processes and responds to other captured material. In some embodiments, keywords can be nested or codewords can be superimposed. The system can have actions related to “IBM data”, “data server”, and “data”, and when the user captures the phrase “IBM data server”, some or all of these keywords You can call actions related to.

  An example of a keyword is the term “IBM”, and the occurrence of that keyword in a document can be associated with directing the reader's web browser to the IBM website. Other examples of keywords are the phrase “Sony Headset”, product model number “DR-EX151”, and the book title “Learning Bash Shell”. The actions associated with these keywords are described in Amazon. querying a list of objects for sale at com, matching one or more of the terms included to one or more objects for sale, and prompting the user for these objects via Amazon Can provide an opportunity to purchase.

  In some embodiments, the system identifies the corresponding electronic document based on the text capture and then performs an action based on the identification (such as presenting an advertisement). For example, the capture of the text “DR-EX151 specification” can identify the product specification document for that product model. In this example, the system retrieves an electronic version of the document and presents the document to the user along with the associated advertisement. The system can present the advertisement separately from the document (such as by sending an email message that provides information related to a similar product), or Advertisements can be presented in documents.

  Some embodiments of the disclosed system perform contextual operations in response to data capture from the rendered document. A contextual action presents a menu of user choices in the context of or in response to other information, such as information in or near text captured from a particular location in the rendered document, or This means that an operation such as an advertisement message is started or started.

  One type of context action is a context advertisement, which means presenting the user with an advertisement selected based on the captured information and some context. A subset of context advertisements, referred to herein as “dynamic context advertisements”, includes dynamically selecting one of several available advertisement messages to present for relevant content.

  Contextual advertising is particularly effective because the advertising message is delivered when people interested in the advertiser's product are examining their interest. Dynamic contextual advertising is particularly effective because it retains the flexibility to present advertising messages that were not available when the content was created or published when the content is being read.

  Various embodiments provide contextual behavior for rendered documents. Context actions can provide actions and responses appropriate to a particular context, i.e., actions can change in response to context changes. An example of contextual action in the system is a menu that appears on a display associated with the portable capture device 302 when a user captures text from a document. This menu can change dynamically depending on the text captured, the location where the text was captured, etc.

  The action can optionally include a verb such as “display” and an object such as “advertisement message”. Additional verbs supported by the system in some embodiments can be sent or received (eg, email message, instant message, capture or copy of document containing keywords), printed (eg, brochure), “browse” ”(For example, a web page), and“ run ”(for example, a computer application).

  In some embodiments, the triggered action includes presenting an advertising message on behalf of an advertiser or sponsor. In some embodiments, an operation can be associated with all documents, a group of documents, a single document, or a portion of a document.

  In some embodiments, the triggered action includes presenting a menu of possible user-initiated actions or selections. In some embodiments, the menu of choices is presented on an associated display device, such as a cell phone display, a personal computer display 421, or a display integrated with the capture device 302, for example. In some embodiments, the menu of choices is also available in whole or in part when the user reviews the capture later from the user account history or life library. In some embodiments, the menu of actions is determined by a markup data and / or markup process associated with a keyword, a rendered document, or a larger group or class of documents.

  In some embodiments, the menu of actions can optionally have zero, one, or more default actions. In some embodiments, the default action is initiated when the user does not interact with the menu, eg, when the user proceeds to a subsequent capture. In some embodiments, the default behavior is determined by markup data and / or markup processes associated with keywords, rendered documents, or larger groups or classes of documents.

  In some embodiments, the action menu is presented such that items that are likely to be selected by the user appear closer to some known location or reference, such as the top of the menu list. The likelihood of selection may be determined in some embodiments by tracking items previously selected by this user of the system and by other users. In some embodiments, the menu of actions can include a subset of standard actions used by the system. Standard menus can appear in various combinations in various contexts, with menu items specific to a particular capture. Some standard actions can appear in the menu if the keyword is not recognized and / or if the capture context is not known. Some standard actions can appear in a menu that is generated when the capture device 302 is disconnected from other components of the system.

Standard operations can include in particular:
・ Speak this word / phrase ・ Translate this into another language (and speak, display, or print)
-Help function-Learn more about this-Display this photo-Bookmark-Underline-Extract (copy)
・ Add to My Calendar ・ Add to My Contact List ・ Purchase this ・ Receive by e-mail ・ Save to My Archive ・ Add voice annotation ・ Play any related voice annotation ・ Associated Display content ・ Display relevant content ・ Find this subject in the content index or table ・ Make note of interest in this topic ・ Go to this website ・ Receive information about this ・ This completed Send the form to me-Close this form for yourself-Submit this form with my information-Search this on the web-Print this document-Make this document on your computer screen or related Upload on the display ・ This word / frame in the document is displayed on your display. Selecting this item, and displays the search for this word / phrase when used in-other contexts to display all occurrences of over's (e.g., multiple choice)
• Extracted into a linear file of notes • Shows that others have written or dictated this document / page / line / phrase • Call this phone number • Notify when this document is available online • Receive information about when / when it is available • Send email to this person / company / address • Get notified if you are the winner of this context / award / offer • This event, award / Lottery / Register in lottery ・ Record reading of this passage ・ Record acceptance of this statement / contract / clause ・ Notify when new information on this topic becomes available ・See topics about you • Get notified when / when this document changes

  In some embodiments, a menu of actions is optionally presented in the vicinity of the content as well as for content specifically captured by the user. In some embodiments, the system uses the choices selected in the previous capture to determine which items to present in subsequent interactions with the document and their order of presentation. Menu items that are frequently selected can be displayed at the top of the menu presentation. In some embodiments, the menu item can optionally invoke additional submenus of related options.

  The following text refers to the notation in the accompanying drawings, described in more detail later. If multiple actions are available for a single keyword, some embodiments of the system use various action rules to select a subset of these actions to perform, for example, A hierarchy can be specified to determine which operations have priority over other operations. For example, the rules can instruct the system to select an action in ascending order of the size of the content body to which the action applies. As an example, if a keyword is captured in a specific chapter of a specific textbook issued by a specific issuer, the system selects the first action associated with the textbook chapter and then enters the specific textbook. The associated second action can be selected, and then the third action associated with all of the textbooks issued by the issuer can be selected. The system also includes the geographic region or location where the capture device 302 is present at the time of capture, the time or date range that the keyword was captured, various other types of context information regarding capture, various types of profile information associated with the user, And / or an action may be selected based on the amount of money or other rewards that the sponsor has agreed to provide the action to the sponsor.

  In some embodiments, the system is integrated into a handheld optical and / or sound capture device 302 wirelessly connected to the computer 212 system, or an acoustic and / or imaging component in a mobile phone, or a PDA (“Personal Digital Assistant”). Utilize handheld optical and / or acoustic capture devices such as similar components.

  In some embodiments, the system includes an optical and / or acoustic capture device 302 that is used to communicate with a keyword server 440 that captures from a rendered document and stores keyword registration information. In some embodiments, the keyword registration information is stored in a database of registered keywords. In some embodiments, this information is stored in a database of markup data. In some embodiments, this information is stored in a markup document associated with the rendered document.

  In some embodiments, the capture device 302 is not a “flatbed” scanner that scans an entire page at a time, but a “pen” scanner that has a scan aperture suitable for scanning text line by line. Portable or handheld scanner. Flatbed scanners are generally not portable and are much larger than pen-type scanners. The pen scanner can have an indicator to indicate to the user when a keyword has been scanned. For example, the scanner can light the LED 332 to inform the user that the scanned word has been recognized as a keyword. The user can start the process by pressing a button on the scanner (or performing a gesture using the scanner), for example, when information related to a keyword is sent to the user Will be done.

  The capture device 302 can have an associated display device. Examples of related display devices include a personal computer display 421 and a display on a mobile phone (216). A menu of actions and other interactions and information data can be displayed on the associated display device. When the capture device 302 is integrated into a mobile phone or uses a mobile phone component, the mobile phone keypad is used to select an option from a menu presented on the mobile phone display and the described system and Functions can be controlled and interacted with.

  If the capture device 302 does not communicate with the keyword server 440 during capture, it has popular keywords, related actions, markup data, and other local caches within the capture device 302, and the capture device 302 is local and independent. To start the operation. An example of a local independent operation is to direct keyword collection, present a selection menu to the user, and receive a user response to the menu. When the capture device 302 then communicates with the keyword server 440, additional information about keywords, markups, etc. is determined and can be operated on.

  In various embodiments, information that associates a word or phrase with an action (eg, markup information) can be communicated in the capture device 302, in a computer 212 system connected to the capture device 302, and / or in the described system. It can be stored in other computer systems. A wide range of devices can be involved as well to perform operations in response to keyword capture.

  In combination with the capture device 302, the keyword server 440 can automatically identify the document from which the text was captured and locate the electronic version of the rendered document. For example, text content in capture can be treated as a document signature. Such signatures typically require 10 or fewer words to uniquely identify the document, with 3 to 8 words being sufficient in many cases. If the additional context information is known, the number of words needed to identify the document can be further reduced. If multiple documents match a single signature, the most likely match (eg, one that includes most captures by that or other users) is specially designated, for example, as the first item in a list or menu Can be presented. If multiple documents match a signature, the previous or next capture is used to remove the ambiguity of the candidate, accurately identify the rendering document owned by the user, and optionally The position of the corresponding digital document can be specified accurately.

  For users who are subscribers to the document retrieval service provided in some embodiments of the system, the keyword server 440 may provide content related to the captured text or the context in which the capture was performed (eg, paragraph, page, The content on the subject of (magazine articles) can be distributed. Thus, the response to the capture may depend dynamically on the context of the capture, and may further depend on user habits and preferences known to the keyword server 440.

  The system enables efficient delivery of electronic content related to text or other information (trademarks, symbols, tokens, images, etc.) captured from rendering publications. The system enables new ways to advertise and sell products and services based on rendering publications such as newspapers and magazines. In conventional newspapers, the news article itself does not include advertisements. This system allows any article text to potentially contain advertisements through the use of keywords associated with products, services, companies, and the like.

  One way in which the system delivers enhanced content for rendering publications is based on the use of keywords in the rendered text. When a preset keyword is captured by the user, distribution of content associated with the keyword is triggered by the captured keyword. In some embodiments, keywords are re-recognized by the keyword server 440 so that content is extracted from the database and sent to a device associated with the user (optionally an output device such as a display or speaker). Become. The associated device can be a nearby display or printer. The system can associate each render keyword (or combination of keywords) with a product or service advertisement. As an example, if a user captures the word “new car” from a rendered document (such as a car magazine), the system triggers to send a local Ford dealership advertisement to a display near the location of the portable capture device 302. can do.

  Similarly, if a user captures a trademark from a rendered document using the capture device 302, the system can send information about the trademark owner's product line to the user. If the user captures the trademark and product name, the information sent to the user will be further narrowed down to provide information specific to that product. For example, if the user captures the word “Sanford”, the system can recognize this word as a trademark of the Sanford office supply company and provide the user with an electronic copy of the catalog of the Sanford office supply (or alternatively). In addition, the system can provide a link to a Sanford web page with an online copy of the catalog). As another example, if the user captures “Sanford Uniball”, the system can be programmed to associate these keywords with a Sanford Uniball ink pen. In this case, the system will deliver to the user information about the Sanford Uniball ink pen line. The system sends this information in the form of an email (with information about the Sanford uniball ink pen or a hot link to a web page with information about the pen) as a push multimedia message to the display near the user. Can be distributed to the user's e-mail account as a catalog sent to the printer.

  This method of associating keywords captured from rendering publications with the delivery of additional content to the user is extremely useful for efficiently providing advertising and other materials to the goal. By identifying keywords captured by the user, the system can provide the user with timely and useful information. The printer manufacturer may pay to advertise the manufacturer's printer that is sent to the user when the user captures the keyword “computer printer”. In addition, certain keyword rights are sold or leased for one or more types of content (eg, within a particular magazine, within articles related to a particular topic, or near other keywords that apply to a topic). be able to. The system can associate the keyword “computer printer” exclusively with a single printer manufacturer, or can associate these keywords with several printer manufacturers (or the topic “computer”). Word keyword “printer” in the context of the article associated with the keyword. If more than one printer manufacturer is associated with a keyword, the system can deliver advertisements, coupons, etc. from each manufacturer (or each manufacturer gets keyword rights in a different context) it can). If the user clicks through to use one of the deliverables or visits the manufacturer's website, the system operator can make a small payment (often called micropayment) to the system operator. You can charge. In some embodiments, the capture device 302 or associated computer 212 can store coupons for later use.

  The system can also further categorize keywords and captures using context about the situation in which the user captured the text. Keywords can be processed separately based on contextual system knowledge / recognition of capture. Examples of contexts include user capture history and interest, other user capture history in the same document, user location, document from which the text was captured, other text or information in the vicinity of the capture (eg, within the same paragraph as the capture) Or on the same page), such as the time of capture. For example, the system can react differently depending on the user's location or depending on the surrounding text in which the keyword appears. By knowing the location of the capture device 302, the service provider can sell or lease the same keyword in different markets. An example would be to sell the same keyword to advertiser # 1 for users in New York and to advertiser # 2 for users in Seattle. Service providers can sell the keyword “hammer” to local tool stores in various cities.

  There are many ways to “lease” or sell keywords in a rendered document. The system splits keyword leases in combination with other keywords (eg, “hammer” that appears near the word “nail” or “construction”) based on the capture time, capture location, and document captured. Can do. As one example of leasing general product descriptions, the keywords “latest book title” and “best seller” can be sold to book sellers. When a user captures the word “latest book title” or “best seller” from a rendered document (such as a newspaper), a list of the best-selling books is sent along with a link to the book seller's web page, which allows the user to Will be able to buy. Alternatively, the link can be a “pass-through” link routed through the keyword server 440 (which allows the system to count and audit click-through transactions), which allows the book seller to click-through. Sales revenue can be separated from system operators, so book sellers can pay for ads on a performance basis (ie, each click-through generated by the service, regardless of sales results). For small payments). Similarly, an advertiser in a printed document can pay based on the advertisement or in the vicinity of the advertisement.

  Incorporation of combined keywords can result in the delivery of different content. For example, capturing the keyword “Hammer” near the keyword “nail” (eg, close in time or number of intervening words) can result in the delivery of advertising content from the tool store. On the other hand, the keyword “Hammer” taken in the vicinity of the keyword “MC” is the entertainer M.C. C. This will result in the distribution of content related to Hammers.

  The trademark owner can use the system to deliver advertisements and messages about the trademark owner's products and services when the user scans the trademark owner's trademark from the rendered document.

  Keyword leases can be divided based on geography. For example, the keyword “buy new car” can be leased nationwide to large car manufacturers and / or leased to local car dealers on a regional basis. If a "new car purchase" is associated with content from a local car dealership, the act of incorporating a "new car purchase" in New York City can result in the delivery of advertisements from a New York City car dealership, The same phrase “buying a new car” captured in Paris will result in the delivery of advertisements from car dealers near Paris.

  Keyword leases can be split based on the document in which the text was captured. For example, by incorporating the keyword “Assault Weapon Ban (attack weapon control law)” from a firearm magazine, distribution of gun support content from the National Rifle Association (National Rifle Association) can be brought about. Incorporating the same keyword “Attack Weapons Control Law” from liberal magazines can result in the distribution of content against handgun violence from Brady Center against gun possession.

  The celebrity name can be used to help celebrities deliver news and messages to fans. For example, the phrase “Madonna” can be associated with content related to the performer Madonna. If the user captures the word “Madonna” from the rendered document, the system will send the concert information for the venue Madonna in the vicinity of the capture location, Amzon. You can send a link to buy Madonna music at com, release of the latest promotions from Madonna marketing company, summary MP3 clips from Madonna's latest hits, and more.

  The cost of associating an advertisement with a particular captured text can vary depending on the capture time. Lease at certain peak costs more and can be less during off-season. For example, the term “diamond” may cost more for a diamond seller to lease during the peak period of the Christmas shopping season than the period in which the annual income tax is paid. As another example, terms such as “lawn mowers” are estimated to have less late night audience (for users capturing text from rendered documents), so late at night than 9 am to 7 pm From 1 am to 5 am may cost less to lease.

  A particular advertisement or message can be associated with many keywords. For example, an advertisement for a Harley Davidson motorcycle can be associated with keywords such as “Harley”, “Harley Davidson”, “new car motorcycle”, “classic motorcycle”.

  An advertisement or message can be associated with a relationship between certain keywords, such as a relative position. For example, if the user captures the word “motorcycle” from the rendered document, and if the keyword “buy” is within 6 words of the keyword “motorcycle”, an advertisement or message related to the motorcycle will be delivered to the user. . If the document context is known, the fact that the keyword “buy” is within a certain distance of the captured word “motorcycle” is known to the system even if only the word “motorcycle” is captured. Thus, an action associated with the keyword “buy motorcycle” can be triggered by capturing only the word “motorcycle” and applying context on the document to further interpret the captured word.

  In some embodiments, the system can generate keywords in response to document identification based on text capture and invoke actions based on the generated keywords. The system can then associate the advertisement with the keyword as described above with respect to the captured keyword. Additional details regarding these embodiments are described below.

  The system is further described below with reference to the accompanying drawings. FIG. 4 is a system diagram illustrating one environment in which the system can operate. The user captures the sequence 401 from the document 400 using the optical and / or acoustic capture device 302. The capture device 302 interacts with a nearby user computer system 212 via a wireless connection, such as IEEE 802.11, 802.16, WLAN, Bluetooth, or infrared connection. Various devices including visual display 421 and renderer 422 can be connected to user computer system 212.

  The capture device 302 passes the captured sequence to the user computer system 212. User computer system 212 sends the sequence to keyword server computer system 440 over network 220 (eg, the Internet or another network). In some embodiments, the keyword server 440 is part of a network of service providers or system operators. In some embodiments, the user computer system 212 is a sequence that allows the system to use additional information such as information identifying the user, information identifying the user's location, information indicating the date and / or time of capture. And select one of several possible actions related to the keyword that the sequence contains.

  As discussed below, the keyword server 440 compares the sequence with a keyword action table 441 (eg, a global markup document) that specifies a particular action for a particular keyword. In some embodiments, the keyword server 440 further uses the document identification index 442 to identify documents based on the captured sequence. As long as the document can be identified, the keyword server 440 accesses a document behavior map 443 (eg, an electronic markup document associated with the rendered document) for the identified document, which map can be identified document or identification. An action to be performed in response to capturing a number of keywords in a particular portion of the rendered document. The keyword server 440 further stores a user profile 444 for the capture device user that contains data about the users available to the system, from alternative actions available to perform on the keywords included in the sequence. You can choose.

  In some embodiments, the action associated with the keyword consists of a verb that indicates the type of action to perform and an object that identifies the content that is the purpose of the action. In some cases, an object can contain actual content, and in other cases, an object can contain an address of actual content or a pointer to actual content. In some cases, the actual content is stored somewhere on the keyword server 440 (eg, another memory location), and in other cases, the content 451 (eg, advertising content) is stored on a separate computer system 450 (advertiser server). Etc.) is stored on.

  While various embodiments have been described with respect to the environment described above, those skilled in the art will recognize that the system may be a single computer system as well as other various similar devices connected in a computer system or in various ways. It will be appreciated that it can be implemented in a variety of other environments, including combinations of:

  FIG. 5 is a flow diagram illustrating exemplary steps performed by the system to perform an action in response to user keyword capture. In step 501, the system receives a sequence captured by a user. In optional step 502, the system identifies the document containing the capture sequence received in step 501 and the location of the capture sequence in this document. At step 503, the system identifies words, phrases, or symbols in the capture sequence where one or more actions are specified. The actions can be specified in a keyword action table, a document action map for the identified document 400, or both. In step 504, the system selects an action associated with the word, phrase, or symbol identified in step 503. In step 505, the system performs the selected operation. After step 505, the system returns to step 501 to receive the next captured sequence.

  One skilled in the art will appreciate that the steps shown in FIG. 5 can be modified in various ways. For example, the order of steps can be rearranged, substeps can be performed in parallel, the illustrated steps can be omitted, or other steps can be included, and so forth.

  FIG. 6 is a table showing sample contents of the keyword operation table. The keyword action table 600 is composed of lines such as lines 601-605, each of which associates a specific action with a specific keyword under certain conditions. Each row is divided into the following columns: a keyword column 611 that includes keywords, and a condition column 612 that includes any conditions that must be met in order to perform a row operation in response to the retrieval of the keywords in the row. , An action verb column 613 containing a row action verb or action type, and an action purpose column 614 containing the purpose of the line action. For example, line 601 shows the following operation when the keyword “pipette” is captured from a document having document ID 01239876: an operation to send an email message from the user of the capture device to the address “info@garlabs.com”. Indicates that can be performed. Line 602 shows a hypertext link with the following behavior: when the keyword “pipette” is imported from a document with document ID 0123432210 or 9766789: the link “Try the Premium Premium Pipette” and the link source “http: // www “.Ilbert.com” can be displayed to the user. Line 603 displays the web page at the following action: “http://www.equips.com/products.htm” when the keyword “pipette” is imported from a document of type “textbook”. Indicates that the action can be performed. Line 604 shows the following behavior when the keyword “Pipe Smith” was captured with a ZIP code from 06465 to 06469 during the time from 6pm to 11pm: label “until 9am tomorrow This indicates that an operation of displaying a hypertext link having “Get a pipe estimate” and a link source “http://www.weblumb.com” to the user can be executed. Line 605 displays a web page with the following behavior when the keyword “Pipe Smith” is captured by a user whose profile shows interest in glass blowing: “http://www.glassshop.com” It indicates that the operation to be displayed can be executed.

  Each of the table diagrams discussed in FIG. 6 and below shows a table designed to make the content and organization more understandable by human readers, but those skilled in the art will need to store this information. The actual data structures used by the system are illustrated, eg, can be organized in different ways, can contain some information other than those shown, can be compressed and / or encrypted, etc. It will be appreciated that this table may be different.

  FIG. 7 is a table showing sample contents of a document action map for a specific document. The document action map is composed of lines such as lines 701-703, each associated with a particular keyword when a particular action is captured in a particular part of the document. Each row is divided into the following columns: a character range column 711 that identifies a range of character positions to which the row is applied, a keyword column 712 that includes keywords, an action verb that includes a row action verb or action type. It is divided into a column 713 and an operation purpose column 714 containing the purpose of the operation of the row. For example, line 701 shows that if the keyword “pipette” is captured anywhere in the character range 1-15120 in the document that is the title of the behavior map, the following behavior: It shows that the operation of displaying “all of needs” to the user can be executed. Line 702 displays the following behavior when the keyword “pipette” is captured somewhere in the text range 50-495 of the document that is the title of the behavior map: “http://www.sanlabs.com/ This indicates that the operation of displaying the web page at “hardened_pipette20.htm” can be executed. Line 703 displays the following behavior when the keyword “Litmus” is captured anywhere in the character range 600-1700: “For a printer located near the user, such as a printer connected to the user computer system 212” It shows that an operation of printing a pamphlet taken out from “http://www.hansen.com/testkit.pdf” can be executed.

  FIG. 8 illustrates exemplary steps performed by the system to perform an action in response to user capture of material unrelated to a keyword, or as an additional process in response to a keyword as shown in FIG. FIG. In step 801, the system receives a sequence captured by a user. In optional step 802, the system identifies the document 400 containing the capture sequence received in step 801 and the location of the capture sequence in this document. In step 803, the system identifies markup data or processes associated with document 400, location in the document, or specific data scanned. The action can be specified in the corresponding digital document of the rendered document, another markup document, markup data and a database of instructions. The markup data can be stored on the capture device 302, in memory or storage on a nearby device, or on a server in the described system. In step 804, the system selects an action associated with the markup determined in step 803. In step 805, the system performs the selected operation. After step 805, the system returns to step 801 to receive the next captured sequence.

  In some embodiments, the system generates keywords in response to document identification based on text capture and invokes actions based on the generated keywords. The system can then associate the advertisement with the keyword. The system can treat all keywords in the same way, i.e., the system can associate a captured keyword with a particular action and associate a generated keyword with the same or similar action.

  For example, a user can capture a short sequence of words from a document that is unique to the document (and thus can be used to identify the document) but does not indicate the theme or subject of the document. In this example, the user captures the phrase “I followed quickly and went down” from the document. Using the captured phrase, the system identifies this document as the Harman Melville novel “White Whale”. Analysis of the document content “White Whale” can trigger advertising on the whaling industry, other novels of Harman Melville, and others. Any possible keyword from the captured sequence, “quickly”, “continue”, “after”, or “descend”, can lead to the system displaying relevant advertising messages to the user. Low.

  Furthermore, the system can reveal that the captured phrase is located in the chapter 5 entitled “Breakfast” of the book. In these cases, the advertisement can be directed to the relevant portion of the book, and can include advertisements related to breakfast cereals, brunch locations in the user's neighborhood, and the like. The system can determine the advertisement based on various levels of granularity with respect to a portion of the document. For example, this portion can be as large as a book chapter (or the entire book as discussed above), or it can be one or two sentences of a section.

  Thus, although the captured text can identify rendered documents scanned by the user, the system may not require the captured text to trigger the advertisement. In addition, although the user does not capture any keywords that will trigger an associated advertising message, the system can provide dynamic advertisements that attempt to meet the user's needs or interests.

  FIG. 9 is a flow diagram illustrating exemplary steps performed by the system to present an advertisement in response to document identification. In step 901, the system receives a sequence captured by a user. In step 902, the system identifies the document 400 that contains the capture sequence received in step 901 and the location of the capture sequence in this document. In step 903, the system identifies the markup data or process associated with the document 400, the identified location in the document, or the particular data scanned. Advertisements can be specified in the corresponding digital document of the rendered document, another markup document, markup data, and a database of instructions. The markup data can be stored on the capture device 302, in memory or storage on a nearby device, or on a server in the described system.

  In optional step 904, the system associates keywords with the identified document, such as the content of the identified document. The associated keywords can be generated by the system or can be pre-associated with the identified document before the document is identified or the text is captured by the user.

  The system can generate keywords for documents identified in several ways. For example, the system can parse the text of the document and perform an iterative process to determine one or more keywords for the document. Word frequency (or no frequency) can be considered. The system can focus on the metadata associated with the document. For example, the document can provide keywords in the metadata in advance, or it can provide a reference for the document. In addition, the system can perform reverse document frequency analysis, vocabulary and semantic analysis, and / or document summarization techniques when analyzing document content.

  In addition, the system can analyze other capture sequences that have identified the document and extract keywords from the previous sequence. The system can utilize any number of techniques in generating keywords and / or associating with identification documents.

  In step 905, the system selects an advertisement associated with the markup determined in step 903. This can include selecting an advertisement using the context. Examples of contexts include the type of rendered document (rendered documents that are novels, articles, questionnaires, etc.) and the source of the rendered document (rendered documents provided by newspapers, magazines, textbooks, displayed web pages, etc.). Other types of context may include user attributes such as the user's identity, the user's location, the group of people the user is a member of, etc.

  In step 906, the system presents the selected advertisement. In some embodiments, the system presents the advertisement in the identified document. When the user distributes the corresponding electronic document of the rendered document that captures the text, the system can add one or more advertisements to the electronic version. The advertisement can be unique to the rendering document, specific to each delivered electronic version, and / or specific to the user. For example, a British user can capture text from an article titled “World Cup 2006 Highlights” and a US user can capture text from the same article. The system can deliver electronic versions of articles to British users with advertisements for British football teams (such as advertisements for London soccer shops), and US users can receive advertisements for US teams (online You can receive an electronic version that includes advertisements for jerseys sold by the shop. In these cases, the system can provide different advertisements in different copies of the same electronic document.

  In some embodiments, the system presents advertisements to the user on the user's scanner, on a nearby display device, on the user's mobile phone, and the like. The system can present advertisements simultaneously with the scan. For example, using the above example, a British user can receive an advertisement for a soccer shop on the user's computer while simultaneously capturing text from an article in “World Cup 2006 Highlights”.

  Alternatively, the system can send an email or other message containing the relevant advertisement to the user (such as the user's email address). The system can present advertisements to the user in many other ways discussed herein.

  After step 906, the system returns to step 901 to receive the next capture sequence.

  In some embodiments, if the system determines that the text capture does not include words associated with the advertising action, it calls some or all of the steps of routine 900 to determine the keywords associated with the document. Then, an advertisement related to text capture can be presented.

  In some embodiments, a method for processing text captured from a rendered document is incorporated into a computing system. In such an embodiment, the system receives a sequence of one or more words that are optically or acoustically captured by a user from a rendered document. The system identifies among the words in the sequence the word or phrase with which the action is associated. The system performs relevant actions for the user.

  Some embodiments incorporate a system for finding keywords in text captured from a rendered document. In such an environment, the system includes a handheld optical and / or acoustic capture device 302 that can be used by a user to capture a sequence of one or more words from a rendered document. The system further includes a processor that identifies among the words of the sequence captured using the handheld optical and / or acoustic capture device 302, the word with which the operation is associated and performs the associated operation on the user.

  Some embodiments incorporate one or more computer memories that store data structures that map keywords to actions. In some embodiments, the data structure specifies, for each of a plurality of keywords that can be captured from the rendered document using handheld optical and / or acoustic capture device 302, an action to be performed on the keyword. It contains entries that contain information to be

  Some embodiments incorporate a system for associating an advertisement with a corresponding electronic document of a rendered document. In such an embodiment, the system identifies the corresponding electronic document, optionally generates keywords associated with the corresponding electronic document, and generates advertisements or optionally generated keywords associated with the corresponding electronic document. Present.

  Some embodiments incorporate a system for finding keywords on the display of the rendered document to the user. In such an environment, the system includes a handheld optical and / or acoustic capture device 302 that can be used by a user to capture a sequence of one or more words from a rendered document via a display. Alternatively, the user can capture the displayed keywords using a cursor or other display icon that is manipulated on the display by the user, such as with a mouse or other pointing device, or a keyboard. The system further includes a processor that identifies the keyword with which the action is associated from among the captured sequence of words and performs the associated action on the user.

  For example, the system may include a presentation layer and capture client that displays content to the user on a web page, in a word pro document, in a PDF document, as an image, or in other graphical or text formats. Rather than trying to design an interface to each of these content formats, in some embodiments, the system captures a display that represents the content and captures using optical character recognition (OCR) technology. Depends on the conversion of the image to text. Specifically, all or part of the viewing device screen buffer used by the user is captured by the system. The contents of the screen buffer are provided to the OCR component to process the captured image and generate the corresponding text for every character included in the image. The system automatically maps any content related to advertisements or other actions to the OCR text identified by the system. In this way, the system can provide advertisements and other actions based on the displayed content regardless of the content format.

  In some embodiments, the presentation layer display client is enabled to superimpose advertising messages on any content regardless of format. When the user views content on the viewing device, all or part of the viewing device's screen buffer is captured by the system. The contents of the screen buffer are supplied to the OCR component, which processes the captured image and generates the corresponding text for every character contained in the image. The system identifies one or more text fragments in the captured text and sends a text fragment schema to the content server. The system compares the received text fragment with the stored text fragment and identifies any stored text fragment that matches the received text fragment. An advertisement corresponding to the matched text fragment is identified for each function and transmitted to the display client. The display client identifies the appropriate location of the advertisement based on the location of the matched text fragment and displays the advertisement in a translucent layer that is overlaid on the content the user is viewing. In this way, the advertisement can be displayed to the user on any content regardless of the format of the content being viewed.

  Accordingly, some embodiments incorporate a system that presents an advertisement in response to capturing an image from a rendered document displayed to a user via a display device. In some cases, the displayed rendered document is identified and presented to the user as discussed with respect to some or all of the steps of FIG.

  The system can receive text fragments captured by the user from the printed version of the rendered document, identify the electronic version of the rendered document, and present the rendered document to the user via a display device associated with the user. The system presents the rendered document as discussed above, captures at least a portion of the display (such as the portion highlighted or selected by the cursor with the user), and optical character recognition (OCR) as described herein. ) Technology can be used to convert the captured image to text and match some or all of the text from the captured image with actions such as advertisements displayed to the user.

  Thus, the system provides an environment that indicates that the user is using a rendered document and presents advertisements regardless of whether the user continues to interact with the printed version of the rendered document or the displayed version of the rendered document. And / or facilitate the execution of operations. The system can receive user selections from the presented version (printed or digital) of the rendered document and provide rich interaction capabilities to the user.

  For example, the system presents an electronic version of a newspaper article on string theory via a display associated with the user. In some cases, the user may have a printed version of a newspaper article, in which case the system is instructed to display the electronic version to the user by capturing a text sequence from the printed version of the newspaper. In this embodiment, the user continues to read articles via the electronic version and no further text is captured from the printed version. In some cases, the user uses the cursor in the electronic version to select the word or phrase of interest (eg, the user selects the phrase “Princeton University Researcher”).

  In response to receiving the selection, the system receives an image of the captured phrase, performs OCR on the image to extract one or more text sequences, and extracts one or more texts. An advertisement associated with the sequence can be identified. In this example, the system determines that the captured image includes the keywords “Researcher” and “Princeton University,” and presents the user with books published by Princeton and advertisements for online courses and seminars that can be studied at Princeton University. To do. Thus, the system continues to provide purpose-based, keyword-based, and / or location-based advertisements to the user even when the user is no longer interacting with the printed version of the document.

CONCLUSION Those skilled in the art will appreciate that the system described above can be easily adapted or expanded in various ways. For example, the system can be used in connection with a wide range of hardware, documents, operation types, and storage and processing schemes. Although the foregoing description refers to various embodiments, the scope of the present invention is defined only by the appended claims and the elements recited in the claims.

Claims (31)

A method in a computing system for responding to text capture comprising:
Receiving a sequence of one or more words optically or acoustically captured by a user from a rendered document;
Identifying an electronic version of the rendered document based on the captured text;
Selecting an advertising message based on the identified electronic version of the rendered document;
Presenting the selected advertising message to the user;
Including methods. The rendering document is sponsored by a third party;
The method according to claim 1. Presenting the advertisement message to the user comprises presenting the advertisement message in an electronic copy of the rendered document.
The method of claim 1. In response to identifying an electronic version of the rendered document, further comprising generating a keyword obtained from analysis of the electronic version, wherein the selecting step is based on the generated keyword.
The method of claim 1. The selection step is based at least in part on the type of the rendered document;
The method of claim 1. The selection step is based at least in part on the source of the rendered document;
The method of claim 1. The selection step is based at least in part on the attributes of the user;
The method of claim 1. The selection step is based at least in part on the identity of the user;
The method of claim 1. The selection step is based at least in part on the location of the user;
The method of claim 1. The selection step is based at least in part on a group of people of which the user is a member;
The method of claim 1. The advertising message is associated with an identified electronic version of the rendered document after the rendered document is created;
The method of claim 1. The advertising message is associated with the identified electronic version after the rendering document is rendered;
The method of claim 1. The action is associated with an identified electronic version of the rendered document for the rendered document only;
The method of claim 1. The step of selecting an advertising message
Analyzing the identified electronic version of the rendered document using word frequency analysis, inverse document frequency analysis, vocabulary and semantic analysis, or document summarization techniques,
The method of claim 1. The advertising message is selected based on content that does not include a capture sequence of one or more words;
The method of claim 1. A system for responding to ingestion of text from a rendered document,
A receiving component that receives a sequence of captured text from the rendered document;
An identification component that identifies an electronic version of the rendered document that contains the captured text;
A content analysis component for analyzing the identified electronic version of the content;
A selection component for selecting an advertisement based on the content analysis;
Including system. The content analysis is one or more of word frequency analysis, inverse document frequency analysis, vocabulary and semantic analysis, or document summarization techniques;
The system of claim 16. The content analysis includes analyzing the type of the rendered document, the source of the rendered document, or the location of the captured text within the rendered document;
The system of claim 16. The content analysis comprises analyzing the identity of a user of the system, the location of the user, or a group of people of which the user is a member;
The system of claim 16. The content analysis component is
A keyword generation component that generates one or more keywords associated with the identified electronic version;
The system of claim 16. One or more computer memories that collectively include an electronic version of the rendered document connected to the advertisement, the data structure including an electronic version of the rendered document, the electronic version captured by the user The electronic version is selected based on a determination that it includes a text sequence, and the advertisement is selected based in part on an analysis of the electronic version of the content;
A computer memory characterized by that. A method for displaying a selected advertisement to a user, comprising:
Receiving data indicating a sequence of text from the rendered document;
Analyzing the received data to identify a corresponding electronic document for the rendered document;
Processing at least a portion of the content of the identified corresponding electronic document to associate one or more advertisements with the corresponding electronic document;
Presenting one or more of the associated advertisements to the user;
Including methods. The processing step includes selecting one of the associated advertisements based on contextual information about the rendered document and presenting the selected related advertisement to the user;
23. The method of claim 22, wherein: The processing step includes selecting one of the associated advertisements based on contextual information about the user;
The method of claim 22. The processing includes determining a keyword representing content of the corresponding electronic document and associating one or more advertisements with the keyword;
The method of claim 22. The presenting step includes presenting an advertisement in the corresponding electronic document;
The method of claim 22. The presenting step includes presenting an advertisement on a display device associated with the user;
The method of claim 22. Retrieving an image to be displayed on a dynamic display device;
Performing optical character recognition on the retrieved image to obtain a sequence of text indicated by the received data;
Further including
The method of claim 22. A computer-readable medium that causes a computing system to perform a method for displaying an advertising message with content, comprising:
The method comprises
Retrieving an image displayed on the identified dynamic display device;
Performing optical character recognition on the retrieved image to obtain a sequence of text;
Identifying an advertising message associated with at least a portion of the obtained sequence of text;
Displaying the identified advertising message;
A computer readable medium comprising: The identified advertising message is displayed on the identified dynamic display device;
30. The computer readable medium of claim 29. The identified advertising message is displayed on a dynamic display device other than the identified dynamic display device that is viewable by a user of a computing system;
30. The computer readable medium of claim 29.

Images