JP2009506394A - Method and machine-readable medium in mixed media document system - Google Patents

Abstract

A mixed media reality (MMR) system and related techniques are disclosed. The MMR system provides a mechanism for forming a mixed media document that includes at least two types of media (eg, printed paper as a first medium and digital content and / or web links as a second medium). In one particular embodiment, the MMR system includes a content-based search database built with an index table, and in a manner that allows searching using a text-based index in a two-dimensional manner between objects retrieved from a printed document. The index table expresses the geometric positional relationship of. A group of rated documents, pages and locations can be calculated from the index table under given data. This technique efficiently converts the features detected in the image patch into text terms (or other searchable features), which are both the features themselves and their geometric positional relationships. Express. Storage means can be used to store additional features for each document image patch.

Description

  The present invention relates to a technique for generating a mixed media document formed from at least two types of media, and in particular, a mixed media reality (MMR: Mixed Media) that combines a print medium and an electronic medium to create a mixed media document. Reality) related to the system.

  The technology for printing and copying documents has been used in many situations for many years. For example, printers and copiers are used in private and professional office environments, in home environments with personal computers, and in document printing and publishing service environments. However, printing and copying techniques have not previously been considered as a means of bridging the gap between static print media (i.e., paper documents) and the interactive "virtual world". . The interactive virtual world includes digital communications, networking, information provision, advertising, entertainment and electronic commerce.

  For centuries, print media has been the primary source of information such as news and advertising information. Over the last few years, the advent and dissemination of personal electronic devices and personal computers such as personal digital assistant (PDA) devices and cellular telephones (e.g. camera-equipped cellular telephones) have made it possible to read print media electronically. The concept of print media was expanded by making it a searchable format and by introducing interactive multimedia capabilities. These are unmatched by conventional print media.

  Unfortunately, there is a gap between the electronically accessible virtual multimedia-based world and the physical world of print media. For example, almost everyone in developed countries has routine access to print media and electronic information, but print media and personal electronic device users are required to form a link between the two (ie, both). Does not have the tools and techniques to encourage the creation of mixed media documents.

  In addition, ordinary print media have the particularly excellent properties of being touchable, requiring no power, and providing structure and preservation permanence, which are inherent to virtual or digital media. Absent. Similarly, ordinary digital media have particularly good properties such as portability (eg, can be carried on a cellular phone or laptop storage) and ease of transmission (eg, email).

  For these reasons, the problem of the present invention is to provide a technique that makes available the advantages of both print media and virtual media.

  At least one form of one or more embodiments of the present invention provides a computer-used method for organizing and accessing information in a mixed media document system. The method includes generating an electronic representation of a paper document, identifying features on the paper document, capturing a two-dimensional aspect of the paper document, identifying a position of the feature, Indexing said features, thereby generating an indexing (index) table. The method may include a preliminary step of receiving the paper document. The method may include storing one or more features associated with at least one of the features. In that case, the one or more features include text information extraction, graphic information extraction, process execution, command execution, ordering, video extraction, sound extraction, information storage One or more actions including at least one of: creating a new document, printing the document, and displaying the document. In certain other cases, the step of identifying features on the paper document and capturing the two-dimensional aspect is aligned horizontally (adjacent or not) or aligned vertically (adjacent). Including the step of identifying objects (with or without). In certain other cases, identifying features on the paper document and capturing a two-dimensional aspect includes identifying horizontal and vertical word pairs (full or partial word pairs). In another particular case, the step of generating an electronic representation of the paper document is performed during the scanning or printing process. In another particular case, the step of identifying features on a paper document and capturing a two-dimensional aspect of the paper document examines the sequence of text by examining the amount of vertical overlap between two successive sequences. Grouping into logical lines. The method responds to a query term based on receiving one or more query terms (query terms) (capturing a two-dimensional positional relationship between objects in the document to be searched) and data from an index table. Computing potential location candidates and at least one mixed media document. In some such cases, receiving one or more query terms that capture a two-dimensional positional relationship between objects in the search target document includes receiving the search target document, at least the search target document. This is done after generating an image of the patch and generating one or more query terms based on the image. In some such cases, generating one or more query terms based on an image includes generating horizontal and vertical word pairs extracted from the image. In another particular case, the step of calculating at least one mixed media document and location candidate includes identifying the location of the stored page that most likely matches the patch of the document to be searched and is most likely to be the center of the patch. Including calculating a location in the page. In some such cases, each word pair is associated with an inverse document frequency, and the step of locating the stored page that most likely matches at least the patch of the searched document appears as a word pair. The accumulator indexed on the document page includes the inverse document frequency function for each word pair. In response to the maximum value exceeding the threshold at the accumulator, the method outputs the associated document page as matching the patch. In such a case, the step of calculating the location in the page that is most likely to be the center of the patch is the step of adding a weight to each cell in the zone around each word pair (the weight for each cell is, for example, , Which can be determined by the product of the inverse document frequency function of the word pair and the normalized geometric distance between the center of the zone and the cell.) And the accumulator corresponding to the cell with the maximum value Includes searching for an Accum array. In response to the maximum value exceeding the threshold, the method further includes reporting the candidate cell as a patch location. In another particular case, the step of calculating at least one mixed media document and location candidate searches each of the one or more query terms in the index table and retrieves one or more locations (locations) associated with each query term. And for each confirmed location, confirming one or more candidate regions including that location. In some such cases, the step of calculating at least one mixed media document and location candidate includes identifying one of the one or more region candidates that most closely matches all of the one or more query terms. In response to determining one of the one or more region candidates that meet a predetermined match criterion, the method continues to confirm that the region matches the search target document.

  At least one other form of one or more embodiments of the present invention includes machine-readable media encoded with instructions (eg, compact disc, diskette, server, memory stick, hard drive, ROM, RAM, etc.). Or any type of media suitable for storing electronic instructions) that, when executed by one or more processors, organizes information in a mixed media document system. And processing to access information is executed by the processor. The process may be similar to or different from the above method, for example.

  At least one other form of one or more embodiments of the present invention provides a method for accessing information in a mixed media document system. The method includes receiving one or more query terms (capturing a two-dimensional positional relationship between objects in the document to be searched) and an index table (indexing document features and feature locations of mixed media documents). Calculating location candidates and at least one mixed media document that may be responsive to a query term based on data from. In some such cases, receiving one or more query terms that capture a two-dimensional positional relationship between objects in the search target document includes receiving the search target document, at least the search target document. This is done after generating an image of the patch and generating one or more query terms based on the image. In some such cases, generating one or more query terms based on an image includes generating horizontal and vertical word pairs extracted from the image. In another particular case, the step of calculating at least one mixed media document and location candidate includes identifying the location of the stored page that most likely matches the patch of the document to be searched and is most likely to be the center of the patch. Including calculating a location in the page. In some such cases, each word pair is associated with an inverse document frequency function, and the step of locating the stored page that most likely matches at least the patch of the document being searched is indexed by the document page in which the word pair appears. Adding an inverse document frequency function for each word pair to the accumulator being processed. In response to the maximum value exceeding the threshold at the accumulator, the method outputs the associated document page as matching the patch. In such a case, the step of calculating the location in the page that is most likely to be the center of the patch is the step of adding a weight to each cell in the zone around each word pair (the weight for each cell is, for example, , Which can be determined by the product of the inverse document frequency function of the word pair and the normalized geometric distance between the center of the zone and the cell.) And the accumulator corresponding to the cell with the maximum value Includes searching for an Accum array. In response to the maximum value exceeding the threshold, the method further includes reporting the candidate cell as a patch location. In another particular case, the step of calculating at least one mixed media document and location candidate searches each of the one or more query terms in the index table and retrieves one or more locations (locations) associated with each query term. And for each confirmed location, confirming one or more candidate regions including that location. The step of calculating at least one mixed media document and location candidate includes identifying one of the one or more region candidates that most closely matches all of the one or more query terms. In response to determining one of the one or more region candidates that meet a predetermined match criterion, the method continues to confirm that the region matches the search target document.

  At least one other form of one or more embodiments of the present invention includes machine-readable media encoded with instructions (eg, compact disc, diskette, server, memory stick, hard drive, ROM, RAM, etc.). Or any type of media suitable for storing electronic instructions) that, when executed by one or more processors, organizes information in a mixed media document system. And processing to access information is executed by the processor. The process may be similar to or different from the above method, for example.

  At least one other form of one or more embodiments of the present invention includes machine-readable media encoded with instructions (eg, compact disc, diskette, server, memory stick, hard drive, ROM, RAM, etc.). Or any type of media suitable for storing electronic instructions) that, when executed by one or more processors, organizes information in a mixed media document system And processing to access information is executed by the processor. The process may be similar to or different from the above method, for example.

  The features and advantages described in this application are not exhaustive and, in particular, many additional features and advantages will be apparent to those of skill in the art upon understanding the following description and drawings. Furthermore, it should be noted that the specific language used herein has been selected primarily for purposes of understanding and description and is not intended to limit the scope of the invention.

  The invention will now be described by way of non-limiting examples in conjunction with the accompanying drawings, in which like numerals are used to refer to like elements.

A Best Mode Mixed Media Reality (MMR) system and associated method embodying the present invention is disclosed. The MMR system provides a mechanism for creating a mixed media document containing at least two types of media, for example, using printing paper as the first medium and digital photos, digital movies, digital audio files, digital text files or web as the second medium. A link or the like may be used. In addition, MMR systems and / or technologies can be used to encourage various business models, combining portable electronic devices (eg, PDAs or cellular phones with cameras) and paper documents to provide mixed media documents. You may consider doing it.

  In one particular embodiment, the MMR system includes a content-based search database that is geometrically between objects extracted from a printed document in a manner that allows search using a text-based index. Represents a two-dimensional positional relationship. Evidence accumulation combines the frequency of appearance of features with the likelihood of a place in a two-dimensional region. In one such embodiment, the MMR database system includes an index table and receives a description calculated with an MMR feature extraction algorithm. The index table identifies xy coordinates, pages and documents within the page where each feature appears. The evidence accumulation algorithm calculates a hypothetical location under the data from the index table, a ranked group of pages and documents. An associated database (or other suitable storage device) can be used to store additional features as needed for each of the documents, pages, and locations.

  The MMR database system may include other components, for example, an MMR processor, a capture device, communication means, memory (including MMR software), and the like. The MMR processor may be coupled to a media type storage or source, to an input device, and to an output device. In one such embodiment, the MMR software includes routines that can be executed by the MMR processor, accessing the MMR document with additional digital content, creating or modifying the MMR document, and using the document for other processing. (Business processing, data search, report, etc.) are executed.

Overview of MMR System Referring to FIG. 1A, a mixed media reality (MMR) system 100a according to one embodiment of the present invention is shown. The MMR system 100a includes an MMR processor 102, a communication means 104, a capture device 106 having a portable input device 168 and a portable output device 170, a memory including the MMR software 108, a base media storage 160, an MMR media storage 162, an output device 164, and An input device 166 is included. The MMR system 100a provides a method of using information from an existing print document (first media type) as an index of a second media type (eg, audio, video, text, update information, service, etc.). Create a mixed media environment.

  The capture device 106 can generate a representation of a printed document (eg, an image, a graphic, or other representation) that is sent to the MMR processor 102. The MMR system 100a matches the representation with the MMR document and other second media types. The MMR system 100a is responsible for actions that respond to input and recognition of expressions. The operations performed by the MMR system 100a may be of any type, for example, extracting information, arranging in a certain order, searching for video or sound, storing information, creating a new document , Printing a document, displaying a document or image, and the like. By utilizing the content-based search database technology described above, the MMR system 100a provides a mechanism for converting printed text into dynamic media, and provides the user with input points for the electronic content and values of the service of interest.

  The MMR processor 102 processes data signals and may include various arithmetic architectures, which execute an instruction set with a complex instruction set computer (CISC) architecture, a reduced instruction set computer (RISC) architecture, or a combination. Including architecture etc. In one particular embodiment, the MMR processor 102 is equipped to perform operations according to the present invention, such as arithmetic logic, microprocessors, general purpose computers, and other information equipment. In other embodiments, the MMR processor 102 comprises a general purpose computer having a graphical user interface, such as a Java (such as a Java operating on an operating system such as a Windows® or Unix® based operating system). It may be generated by a program written in Java (registered trademark). Although only one processor is depicted in FIG. 1A, multiple processors may be included. The processor is coupled to the MMR 108 and executes instructions stored therein.

  The communication means 104 is any device or system that couples the capture device 106 to the MMR processor 102. For example, the communication means 104 can be a network (e.g., WAN and / or LAN), a wired link (e.g., USB, RS232 or Ethernet), a wireless link (e.g., infrared, Bluetooth or 802.11), a mobile communication link (e.g., GPRS or GSM), public switched telephone network (PSTN) links, or some combination thereof. Various communication architectures and protocols can be used here.

  The capture device 106 may be any device capable of digitally capturing images or data via an input device 168, including means such as a transceiver that interfaces with the communication means 104. The capture device 106 may optionally include an output device 170, or may be portable. For example, the capture device 106 can be a standard camera cellular phone, a PDA device, a digital camera, a barcode reader, a wireless IC tag (RFID) reader, a computer peripheral such as a standard webcam, or a PC An embedded device such as a video card may be used. Several specific examples of the capture device 106 are each described in detail with reference to FIGS. 2A-2D. Furthermore, the capture device 106 may be a software application, which enables content-based searching and links the capture device 106 with the infrastructure of the MMR system 100a / 100b. A more detailed function of the capture device 106 will be described with reference to FIG. 2E. Various general and customized capture devices 106 and their functions and architecture will be apparent from this application.

  The memory 108 stores instructions and / or data executed by the processor 102. The instructions and / or data are comprised of code that performs any and / or all of the above techniques. The memory 108 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, or any other suitable memory device. The memory 108 is described in further detail below with reference to FIG. In certain embodiments, memory 108 includes a set of MMR software, an operating system, and other application programs (eg, word processing applications, email applications, financial applications, and web browser applications).

  Base media storage 160 is for storing secondary media types in their original form, and MMR media storage 162 is for MMR documents, databases and other information (detailed information for creating an MMR environment). Is for storing. Although shown separately in the figure, in other embodiments, the base media storage 160 and MMR media storage 162 may be part of the same storage device or may be integrated. The data storage 160, 162 stores instructions and data of the MMR processor 102 (hard disk drive, floppy disk drive, CD-ROM device, DVD-ROM device, DVD-RAM device, DVD-RW device, flash memory device or One or more devices (such as any other suitable mass storage device, etc.) may be included.

  The output device 164 is operatively coupled to the MMR processor 102 and represents any device (device that displays, produces sound, or presents content) that is equipped to output data. For example, the output device 164 may be any of various types such as a printer, a display device, and / or a speaker. An exemplary display output device 164 includes a cathode ray tube (CRT), a liquid crystal display (LCD), any display device provided as well, a screen or monitor, and the like. In one embodiment, the output device 164 is provided on a touch screen, and a touch-sensitive transparent panel covers the screen of the output device 164 on the touch screen.

  The input device 166 is operably coupled to the MMR processor 102, such as a keyboard, cursor controller, scanner, multifunction device, still camera, video camera, keypad, touch screen, detector, RFID tag reader, switch or some other means, etc. Any of a variety of types (some means allows the user to interact with system 100a). In one embodiment, input device 166 is a keypad and cursor controller. The cursor controller includes, for example, a mouse, trackball, stylus, pen, touch screen, pad, cursor direction keys, or other means for moving the cursor. In other embodiments, the input device 166 is a macrophone, audio insertion / expansion card, and may be used to facilitate speech recognition and / or audio processing in general purpose computer systems, analog to digital converters, and digital signal processors. It is designed.

  FIG. 1B shows a functional block diagram of an MMR system 100b constructed in accordance with another embodiment of the present invention. In this embodiment, the MMR system 100b includes an MMR computer 112 (operated by a user 110), a network media server 114, and a printer 116 that generates a print document 118. The MMR system 100b further includes an office portal 120, a service provider server 122, and an electronic display 124, which are electronically connected to a set top box 126 and a document scanner 127. The communication link between the MMR computer 112, the network media server 114, the printer 116, the office portal 120, the service provider server 122, the set top box 126, and the document scanner 127 is provided by a network 128, which is a LAN (e.g. Or a home network), a WAN (eg, the Internet, a corporate network), a LAN / WAN combination, or any other suitable data path through which multiple computing devices may communicate.

  The MMR system 100b further includes a capture device 106, which includes one or more of a network media server 114, a user printer 116, an office portal 120, a service provider server 122, an electronic display 124, a set top box 126, and a document scanner 127. , Wireless communication via cellular infrastructure 132, wireless fidelity (Wi-Fi) technology 134, Bluetooth technology 136 and / or infrared (IR) technology 138, and the like. Alternatively or additionally, the capture device 106 may be wired via a wired technology 140 with an MMR computer 112, a network media server 114, a printer 116, an office portal 120, a service provider server 122, a set-top box 126 and a document scanner 127. Can communicate in the form. Although the Wi-Fi technology 134, the Bluetooth technology 136, the IR technology 138, and the wired technology 140 are shown as separate elements in FIG. 1B, such technologies (e.g., MMR computer 112, network media server) 114, capture device 106, etc.) may be integrated within the processing environment. In addition, the MMR 100b further includes Geo location means 142 to communicate with the service provider server 122 or network 128 wirelessly or wired. This may be integrated into the capture device 106.

  The MMR user 110 is any individual who uses the MMR system 100b. The MMR computer 112 is any desktop, laptop, network computer or other processing environment. The user printer 116 is any home, office, or business printer capable of creating a print document, and the print document is a document composed of one or more print pages.

  The network media server 114 is a network computer and has information and / or applications accessed by a user of the MMR system 100b via the network 128. In certain embodiments, the network media server 114 is a centralized computer on which text source files, web pages, audio and / or video files, image files (eg, still image photos), etc. Various media files such as are stored. The network media server 114 is, for example, a Comcast video on demand service of Comcast, a Ricoh document mall of Ricoh Innovations Incorporated, or a video server of Google Incorporated. In general, the network media server 114 provides access to any data that is attached, integrated, or associated with the print document 118 by the capture device 106.

  The office portal 120 is an optional means for capturing events that occur in the environment of the MMR user 110, such as capturing events that occur in the office of the MMR user 110, for example. For example, the office portal 120 may be a computer separate from the MMR computer 112. In that case, the office portal 120 may be directly connected to the MMR computer 112 or may be connected to the MMR computer 112 via the network 128. Alternatively, the office portal 120 may be incorporated in the MMR computer 112. For example, the office portal 120 may be configured with a regular personal computer (PC) and extended with appropriate hardware to support some associated capture device 106. The office portal 120 may include capture devices such as video cameras and audio recorders. Further, the office portal 120 may capture and store data from the MMR computer 112. For example, the office portal 120 can monitor and receive events and functions that occur on the MMR computer 112. As a result, the office portal 120 can record all audio and video in the physical environment of the MMR user 110 and can record all events that occur on the MMR computer 112. In one particular embodiment, the office portal 120 captures an event—for example, a video screen capture from the MMR computer 112 while a document is being edited. At that time, the office portal 120 captures the viewed website and other documents created when a given document is created. That information may later be made available to MMR user 110 via his / her MMR computer 112 or capture device 106. Furthermore, the office portal 120 may be used as a multimedia server in preparation for clips that users attach to their documents. In addition, the office portal 120 can handle other office events such as conversations that occur while the document is on the desk (eg, telephone or office conversations), telephone conversations, and small meetings in the office. May be captured. A video camera (not shown) in the office portal 120 may verify the document on the physical desktop of the MMR user 110 by utilizing the same content-based search technology developed for the capture device 106. .

  The service provider server 122 is any commercial server that holds information or applications, and the information and the like can be accessed by the MMR user 110 of the MMR system 100b via the network 128. In particular, the service provider server 122 represents any service provider associated with the MMR system 100b. The service provider server 122 may be, for example, without limitation, a commercial server of a cable TV provider such as Comcast Corporation, a cellular telephone service provider such as Verizon Wireless, an Internet service provider such as Adelphia Communications, Sony, etc. Online music service providers such as corporations.

  The electronic display 124 is any display device such as, but not limited to, a standard analog or digital television (TV), flat screen TV, flat panel display or projection system. The set top box 126 is a receiving device that processes signals coming from a satellite antenna, an antenna, a cable, a network, a telephone line, or the like. The manufacturer of the set top box is, for example, advanced digital broadcast. Set top box 126 is electronically connected to the video input of electronic display 124.

  The document scanner 127 is a commercially available document scanner device such as KV-S2026C full-color scanner of Panasonic Corporation. The document scanner 127 is used when converting an existing printed document into an MMR document.

  The cellular infrastructure 132 represents multiple cell towers and other cellular network interconnections. In particular, by utilizing cellular infrastructure 132, two-way voice and data communication can be performed with a handheld, portable and in-vehicle phone via a wireless modem embedded in a device such as capture device 106 or the like. Done.

  The Wi-Fi technology 134, the Bluetooth technology 136, and the IR technology 138 represent technologies for performing wireless communication between electronic devices. The Wi-Fi technology 134 is a technology related to a wireless local area network (WLAN) based on the 802.11 standard or the like. Bluetooth technology 136 is defined in the Telecommunications Industry Standard specification, which describes how cellular telephones, computers and PDAs are interconnected via short-range wireless connections and the like. IR technology 138 enables electronic devices to communicate through short-range wireless signals. For example, IR technology 138 may be a line-of-sight wireless communication medium used in television remote controllers, laptop computers, PDAs and other devices. IR technology 138 uses a mid-microwave spectrum below visible light. Further, in one or more other embodiments, wireless communications may be supported using the IEEE 802.15 (UWB) and / or 802.16 (WiMAX) standards.

  The wired technology 140 is any wired communication means such as standard Ethernet connection or universal serial bus (USB) connection. By using cellular infrastructure 132, Wi-Fi technology 134, Bluetooth technology 136, IR technology 138 and / or wired technology 140, capture device 106 communicates bi-directionally with any or all of the electronic devices of MMR system 100b. it can.

  The geolocation means 142 is any suitable means for confirming the geographical position. The geolocation means 142 is a GPS satellite that provides position data to a terrestrial GPS receiver, for example as is well known. In the example shown in FIG. 1B, location data is provided to a user of the MMR system 100b via a service provider server 122, which is connected to a network 128 that is coupled to a GPS receiver (not shown). Yes. Alternatively, the geolocation means 142 is a group of cell towers (e.g., a portion of the cellular infrastructure 132), such as triangulation means, cell tower identification (ID) means, and / or (such as means for confirming geographic location). Prepare enhanced service 911. Alternatively, the geolocation means 142 may be provided by a signal strength measurement from a known location of the WiFi access point or Bluetooth device.

  In operation, the capture device 106 functions as a client owned by the MMR user 110. There is a software application that enables content-based search processing and capture devices via cellular infrastructure 132, Wi-Fi technology 134, Bluetooth technology 136, IR technology 138 and / or wired technology 140 106 is coupled to the infrastructure of the MMR system 100b. In addition, software applications that enable a number of operations also exist in the MMR computer 112, including but not limited to print capture processing, event capture processing (e.g., storing document editing history), server processing (e.g., Save data and events for later service to other companies) or print management processing (e.g., printer 116 queues data required for MMR, such as document layout and multimedia clips, etc.) May be set to line up).

  A networked media server 114 provides access to data attached to a print document, such as a print document 118, which is printed via the MMR computer 112 belonging to the MMR user 110. In so doing, a second medium, such as video or audio, is associated with the first medium, such as a document. Further details of the software application and / or means for associating the second medium with the first medium are described below with reference to FIGS. 2E, 3, 4 and 5.

Capture Device FIGS. 2A, 2B, 2C and 2D illustrate a capture device 106 according to one embodiment of the present invention. More specifically, FIG. 2A shows a capture device 106a that is a cellular telephone. FIG. 2B shows a capture device 106b which is a PDA device. FIG. 2C shows a capture device 106c, which is a computer peripheral device. An example of a computer peripheral device is any standard webcam. FIG. 2D shows a capture device 106d that is incorporated into a computing device (eg, MMR computer 112, etc.). For example, the capture device 106d is a computer graphics card. Specific details of the capture device 106 will be described with reference to FIG. 2E.

  In the case of the capture devices 106a, 106b, the capture device 106 may be related to possession of the MMR user 110. Their physical location may be tracked by the geolocation means 142 or by the ID number of each cell tower in the cellular infrastructure 132.

  Referring to FIG. 2E, a functional block diagram of an example capture device 106 according to one embodiment of the present invention is shown. The capture device 106 includes a processor 210, a display 212, a keypad 214, a storage device 216, a wireless communication link 218, a wired communication link 220, an MMR software group 222, a capture device user interface (UI) 224, and a document fingerprint verification module 226. , A third party software module 228, and at least one of various capture means 230. Specific examples of the capturing means include, but are not limited to, a video camera 232, a still camera 234, a voice recorder 236, an electronic highlighter 238, a laser 240, a GPS device 242 and an RFID reader 244.

  The processor 210 is a central processing unit (CPU), for example, but not limited to, a Pentium (registered trademark) microprocessor manufactured by Intel Corporation. Display 212 is any standard video display means such as used in handheld electronic devices. More specifically, the display 212 is, for example, any digital display (such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display). Keypad 214 is any standard alphanumeric input means, such as a keypad used in standard computer devices and portable electronic devices (eg, cellular phones). The storage device 216 is any volatile or non-volatile memory device, such as a well-known hard disk drive or random access memory (RAM) device.

  The wireless communication link 218 is a wireless data communication means, which is a direct one as is well known via an access point (not shown) and via a LAN (eg, IEEE 802.11 Wi-Fi or Bluetooth technology). Perform one-to-one communication or wireless communication. The wired communication link 220 is a wired data communication means, and performs direct communication via, for example, standard Ethernet and / or USB connection.

  The MMR software group 222 is overall management software, and executes MMR processing that merges a certain type of media with the second type. Further details of the MMR soft air group 222 will be described with reference to FIG.

  A capture device user interface (UI) 224 is a user interface for operating the capture device 106. By using the capturing device UE224, various menus regarding function options are presented to the MMR user 110. More specifically, the capture device user interface (UI) 224 menu allows the MMR user 110 to manage tasks, including but not limited to interacting with documents and reading data from existing documents. , Writing data to existing documents, viewing and interacting with the enhanced reality associated with those documents, viewing with the enhanced reality associated with the documents displayed on his / her MMR computer 112 And allows interaction and the like.

  The document fingerprint matching module 226 is a software module that extracts features from the captured text image via at least one capturing means 230 of the capturing device 106. The document fingerprint matching module 226 can also perform pattern matching between the captured image and the document database. According to one embodiment at the most basic level, the document fingerprint matching module 226 determines a portion (small section) of an image patch in a large page image, and the large page image is derived from a large collection of documents. Selected. Receiving captured data, extracting a representation of the image from the captured data, performing patch determination and motion analysis in the document, performing a synthesis of the determination, xy coordinates where the input image is located The document fingerprint verification module 226 includes a routine or program such as outputting a list of For example, the document fingerprint matching module 226 combines horizontal and vertical (vertical and horizontal) features extracted from a text fingerprint (fingerprint) image and extracts the features from the document and the document from which the features were extracted. An algorithm for checking the section may be used. Once the features are extracted, a printed document index (not shown), for example, located in the MMR computer 112 or network media server 114 is queried to confirm the symbolic document. Under the control of the capture device user interface (UI) 224, the document fingerprint verification module 226 has access to the printed document index. The printed document index is described in further detail with reference to the MMR computer 112 of FIG. Note that in an alternative embodiment, the document fingerprint verification module 226 may be part of the MMR computer 112 and not be provided in the capture device 106. In such an example, capture device 106 transmits raw capture data to MMR computer 112 for image extraction, pattern matching, document and position determination. In yet another embodiment, the document fingerprint matching module 226 performs only feature extraction, and the extracted features are sent to the MMR computer 112 for pattern matching and authentication.

  Third party software module 228 represents any third party software module that supports any processing that occurs in capture device 106. Examples of Daidan software include security software, image detection software, image processing software, and MMR database software.

  As mentioned above, the capture device 106 may include any number of capture means 230, examples of which are described below.

  The video camera 232 is a digital video recording device and may be found on a standard digital camera or cellular telephone.

  The still camera 234 is any standard digital camera device that can capture digital images.

  The voice recorder 236 is any standard recording device (microphone and associated hardware) that can capture the audio signal and output it in digital form.

  The electronic highlighter 238 is an electronic highlighter that can scan, store and transfer printed text, barcodes and small images to a PC, laptop computer or PDA device. The electronic highlighter 238 is, for example, a Quicklink Pen Handheld Scanner from Withcom Technologies, which stores information in the pen and computer information via a serial port, infrared communication or USB adapter. Can be transferred directly to.

  As is well known, the laser 240 is a light source that generates light by stimulated emission, coherent, or quasi-monochromatic light. For example, laser 240 is a standard laser diode, a semiconductor device that emits coherent light when forward biased. Associated with and included with laser 240 is a detector that measures the amount of light reflected from the image to which laser 240 is directed.

  The GPS device 242 is any portable GPS receiver that prepares position data such as digital data of latitude and longitude. Specific examples of portable GPS receivers 242 include Sony Corporation's UV-U70 portable satellite navigation system, Thales North America Corporation's Magellan brand roadmate series GPS devices, Meridian Series GPS devices and Xpro EXplorist Series GPS device. The GPS device 242 provides a method for determining the position of the capture device 106 in real time, to some extent by triangulation, as is well known to the plurality of geolocation means 142.

  The RFID reader 244 is a commercially available RFID tag reader system, such as the TI-RFID system manufactured by Texas Instruments. The RFID tag is a wireless device for confirming unique items (items) by using a radio wave. As is well known, an RFID tag is formed by a microchip attached to an antenna, and a unique digital identification number is stored in the microchip.

  In certain embodiments, capture device 106 includes processor 210, display 212, keypad 214, storage device 216, wireless communication link 218, wired communication link 220, MMR software group 222, capture device user interface (UI) 224. , A document fingerprint verification module 226, a third party software module 228, and at least one of various capture means 230. In that case, the capture device 106 is a fully functional device. Alternatively, the capture device 106 may have fewer functions or may include only a limited set of functional components. For example, the MMR software group 222 and the document fingerprint verification module 226 exist remotely, for example, the MMR computer 112 or the network media server 114 of the MMR system 100b is captured via the wireless communication link 218 or the wired communication link 220. It may be accessed from device 106.

MMR Computer Referring to FIG. 3, an MMR computer 112 constructed in accordance with one embodiment of the present invention is shown. As shown, the MMR computer 112 is connected to a network media server 114 containing one or more multimedia (MM) files 336, to a user printer 116 that generates a printed document 118, to a document scanner 127, and to a capture device 106. Connected, the capture device includes a first instance of a document fingerprint verification module 226 and a capture device user interface (UI) 224. The communication link between these components may be a direct link or over a network. In addition, document scanner 127 includes a second instance of document fingerprint verification module 226 ′.

  The MMR computer 112 of this embodiment stores one or more source files 310, a first source document (SD) browser 312, a second SD browser 314, a printer device 316, a print document (PD) capture module 318, and a PD index 322. Document event database 320, event capture module 324, document analysis module 326, multimedia (MM) clip browser / editor module 328, MM printer driver 330, document video analysis (DVP: Document-to-Video Parser) printing system 332 And video paper 334.

  Source file 310 represents any source file that is an electronic representation of a document (or a portion thereof). Specific examples of source files 310 are hypertext markup language (HTML) files, Microsoft Word files, Microsoft PowerPoint files, simple text files, portable document format (PDF) files, etc., which are the hard drives of MMR computer 112 ( Or other suitable storage).

  The first SD browser 312 and the second SD browser 314 are plug-ins for stand-alone PC applications or existing PC applications and allow access to data associated with the source file 310. The first and second SD browsers 312 and 314 extract the original HTML file or MM clip for display on the MMR computer 112.

  The printer device 316 is printer driver software and controls the communication link between the application and the page description language (used in any particular printer) or printer control language as is well known. In particular, whenever a document such as the printed document 118 is printed, the printer driver 316 provides the printer 116 with data to correct the control command, such as those prepared by Ricoh Corporation for the printing device. In one embodiment, the printing device 316 automatically captures an xy coordinate, font, and point size representation of all characters of all printed pages, unlike a normal printing device. In other words, the printing device captures information about the contents of the entire document to be printed and feeds the data back to the PD capture module 318.

  A print document (PD) capture module 318 is a software application that captures a printed representation of a document, allowing the layout of character and graphic print pages to be retrieved. Further, by utilizing the PD capture module 318, a printed representation of the document is captured automatically, in real time and / or at the time of printing. More specifically, the PD capture module 318 is a software routine that captures the two-dimensional arrangement of characters on the printed page and transmits the information to the PD index 322. In one embodiment, the PD capture module 318 performs processing by obtaining Windows text layout commands for all characters on the printed page. The text layout command indicates to the operating system (OS) the xy coordinates of all characters on the printed page as well as the font and point size. In practice, the PD capture module 318 intercepts the print data transmitted to the printer 116. In the illustrated example, the PD capture module 318 is coupled to the output of the first SD browser 312 to capture data. Alternatively, the functionality of the PD capture module 318 may be performed directly within the printer driver 316. It will be apparent from the present disclosure that various configurations are possible.

  The document event database 320 is any standard database that has been modified to store relationships between printed documents and events according to one embodiment of the present invention. (The document event database 320 is further described below as an MMR database with reference to FIG. 34A.) For example, the document event database 320 is transferred from the source file 310 (eg, a word file, HTML file, PDF file) to the print document 118. Stores bi-directional links to related events. Examples of events include capturing a multimedia clip at the capture device 106 immediately after the word document is printed, adding multimedia to the document with the client application of the capture device 106, or annotating the multimedia clip. In addition, other events associated with the source file 310 that may be stored in the document event database 320 include logging when a given source file 310 is opened, closed, or deleted, a given source file This includes logging when 310 is in the active application on the desktop of MMR computer 112, logging the number of document “copy” and “move” operations and destinations, logging the editing history of a given source file 310, and so on. Such events are captured by the event capture module 324 and stored in the document event database 320. The document event database 320 is coupled to receive the output of the source file 310 to receive the output of the event capture module 324, PD capture module 318 and scanner 127, and is also coupled to the capture device 106 to receive queries and data, Prepare output.

  The document event database 320 also stores a PD index 322. A PD index is a software application that maps features extracted from images of a printed document to their symbolic forms (eg, scanned images to words). In one embodiment, PD capture module 318 provides PD index 322 with xy coordinates of all characters on the printed page, as well as fonts, point sizes, and the like. The PD index 322 is created when a given document is printed. However, all print data may be captured and stored in the PD index 322 in a manner obtained at a later time. For example, if the printed document 118 includes the word “garden” physically located on the page one line before the word “rose”, the PD index 322 supports such a query function (ie, the word “rose”). The word “garden” precedes “rose”. The PD index 322 contains a record of which page of which page of which document (where in the page) the word “garden” appears before the word “rose”. Accordingly, the PD index 322 is organized to support feature-based or text-based queries. The content of the PDD index 322 is an electronic representation of the printed document, using the PD capture module 318 during the printing process and / or using the document fingerprint verification module 226 'of the document scanner 127 during the scanning process. Is generated. Further database 320 and PD index 322 architectures and functions are described below with reference to FIGS. 34A-C, 35 and 36.

  The event capture module 324 is a software application that captures events at the MMR computer 112 and is associated with a given print document 118 and / or source file 310. These events are captured in the life cycle of a given source file 310 and stored in the document event database 320. In a particular embodiment, event capture module 324 is used to capture events related to an active HTML file in a browser on computer 112, such as first SD browser 312. These events may include the file name of another document that was opened at the same time when the HTML file was displayed on the MMR computer 112 or when the HTML file was displayed or printed. This event information, for example, wanted MMR user 110 to know (at a later time) what document he / she was viewing or working on when the HTML file was displayed or printed. It is particularly useful in cases. Specific examples of events captured by the event capture module 324 include document editing history, video of an office meeting made at a time when a given source file 310 was on the desktop, and the given source file 310 opened. Phone calls (which may be captured at the office portal 120), etc.

  Specific functions of the event capture module 324 are: 1) tracking-tracking active files and applications, 2) keystroke capturing-keystroke capture and association with active applications, 3) frame buffer capturing and indexing Each frame buffer image includes being indexed with optical character recognition (OCR) results of the frame buffer data so that a portion of the printed document can be verified when displayed on the screen. Alternatively, text can be captured along with a graphical display interface (GDI) shadow dll to track commands that draw characters for the PC desktop issued by the PC operating system. The MMR user 110 may point the capture device 106 with a document to see when it was active on the desktop of the MMR computer 112; the above functions include: 4) History capture read-frame buffer capture and The indexing process data is linked to an analysis of when the document was active on his / her MMR computer 112 desktop and tracks how long it is and what part of a particular document. Including making displayable at 110. At that time, in order to infer whether or not the MMR user 110 has read the document, relevance to other events such as keystrokes and mouse movements may be considered.

  The combination of the document event database 320, the PD index 322, and the event capture module 324 may be implemented locally on the MMR computer 112 or may be implemented as a shared database. When implemented locally, the security level is required to be lower than when implemented in a shared format.

  The document analysis module 326 is a software application that analyzes the source file 310 associated with each printed document 118, where useful objects (e.g., uniform resource locator (URL), address, title, author, time or Identify a phrase that represents a place (eg, Harry Dee Building). In doing so, the location of these objects in the printed version of the source file 310 is identified. The output of the document analysis module 326 can be used by the receiving device to extend the presentation of the document 118 with additional information to improve the accuracy of pattern matching. Further, the receiving device retrieves the web page associated with the URL, for example using the location in the case of a URL. Document analysis module 326 is coupled to receive source file 310 and provides an output to document fingerprint verification module 226. Although only shown to be coupled to the document fingerprint verification module 226 of the capture device, the output of the document analysis module 326 may be output to all or any number of the document fingerprint verification modules 226. Can be combined. Further, the output of the document analysis module 326 may be stored in the document event database 320 for later use.

  The MM clip browser / editor module 328 is a software application that provides an authoring function. The MM clip browser / editor module 328 may be a stand-alone application or a plug-in operating on a document browser (represented by a wavy line with respect to the second SD browser 314). The MM clip browser / application 328 is coupled to the network media server to display the multimedia file to the user and receive the multimedia file 336. Furthermore, when the MMR user 110 creates a document (eg, when attaching a multimedia clip to a document), the MM clip browser / application 328 becomes a support tool for that function. The MM clip browser / application 328 is an application that shows metadata such as information analyzed from a document printed when a multimedia document is created.

  The MM printer driver 330 provides a function for creating an MMR document. For example, the MMR user 110 may highlight text generated by the MM printer driver 330 in a user interface (UI) and retrieve multimedia data or perform some other process on the network 128 or on the MMR computer 112 An action containing may be added to the text. The combination of the MM printer driver 330 and the DVP printing system 332 provides an alternative output that utilizes barcodes. This format does not necessarily require content-based search technology. The MM printer driver 330 is a printer driver that supports video paper technology, ie, video paper 334. The printer driver 330 of the MM creates a paper expression including a barcode as a means for accessing the multimedia. On the other hand, the printer driver 316 creates a paper expression including MMR-compatible technology as a means for accessing the multimedia. The authoring function included in the combination of the MM clip browser / application 328 and the SD browser 314 creates a sonar time output format as the SD browser 312 and enables creation of an MMR document in preparation for content-based search. The DVP printing system 332 couples the processing of any data in the document event database 320 associated with the document to the printed representation with an explicit or implicit barcode. Implicit bar codes relate to patterns of text attributes, such as those used for bar codes.

  Video paper 334 is a technique for representing audiovisual information on a printable medium such as paper. In video paper, barcodes are used as indicators for electronic content stored in or accessible without a computer. As the user scans the barcode, the system outputs video clips and other multimedia content associated with the text. Systems exist that print audio or video paper, and these systems essentially provide a paper-based interface for multimedia information.

  The network media server 114 MM file 336 represents any set of various file types and file formats. For example, the MM file 336 is a text source file, a web page, an audio file, a video file, an audio / video file, and an image file (for example, a still picture).

  As described in FIG. 1B, the document scanner 127 is used to convert an existing printed document into an MMR specification document. Still referring to FIG. 3, however, the document scanner 127 applies MMR processing to an existing document by applying the feature extraction processing of the document fingerprint matching module 226 'to all pages of the scanned document. Used to make it possible to do. Thereafter, the scanning and feature extraction results are entered into the PD index 322, and an electronic representation of the scanned document is stored in the document event database 320. The information in the PD index 322 can be used to create an MMR document.

  With further reference to FIG. 3, it should be noted that the software functions of the MMR computer 112 are not limited to the MMR computer 112 alone. Alternatively, the software functions shown in FIG. 3 may be distributed in any configuration determined by the user in the MMR system 100b of the MMR computer 112, the network media server 114, the service provider server 122, and the acquisition device 106. . For example, source file 310, SD browser 312, SD browser 314, printer 316, PD capture module 318, document event database 320, PD index 322, event capture module 324, document analysis module 326, MM clip browser / editor module 328, The MM printer driver 330 and the DVP printing system 332 may be fully contained within the capture device 106, thereby providing enhanced functionality for the capture device 106.

MMR Software Group FIG. 4 illustrates software component groups included in the MMR software group 222 according to one embodiment of the present invention. It should be noted that all or some of the MMR software group 222 may be included in the MMR computer 112, the capture device 106, the network media server 114, and other servers. Further, other examples of the MMR software group 222 may have any number of the illustrated components, from one to all. The MMR software group 222 of this embodiment includes multimedia annotation software 410 (including a text content-based extraction unit 412, an image content-based extraction unit 414, and a steganographic correction unit 416), a paper reading history log 418, , Online reading history log 420, collaborative document checking unit 422, real-time notification unit 424, multimedia acquisition unit 426, desktop video reminder unit 428, web page reminder unit 430, physical history log 432, entry Completed document confirmation unit 434, propagation time unit 436, position confirmation unit 438, PC authoring unit 440, document authoring unit 442, capture device authoring unit 444, unconscious upload unit 446, and document version acquisition unit 448 A PC document metadata unit 450, a capture device user interface (UI) unit 452, a domain specific component 454, including.

  The multimedia annotation software 410 combined with the organization of the document event database 320 forms the basic technology of the MMR system 100b according to a specific embodiment. More specifically, multimedia annotation software 410 is for managing multimedia annotations on paper documents. For example, the MMR user 110 points the capture device at some portion of the paper document and annotates that portion using at least one capture means 230 of the capture device 106. In a specific example, a lawyer directs an annotation (creates an audio file) for a certain part of the contract. The multimedia data (audio file) is automatically attached to the original electronic version of the document. Subsequent printing of the document optionally includes an indication of the presence of those annotations.

  The text content-based extraction unit 412 is a software application that extracts content-based information from text. For example, by using a text content-based extraction unit 412, content is obtained from a text patch (part), the original document and sections in the document are confirmed, or other information linked to the patch It is confirmed. The text content-based extraction unit 412 may use an OCR-based technique. Alternatively, non-OCR-based techniques for performing content-based extraction from text include a two-dimensional array of word lengths within a patch of text. An example of a text content-based extractor 412 is an algorithm that combines horizontal and vertical (vertical and horizontal) features extracted from an image of a portion of text to produce a document and its document. Check the extracted section of. Horizontal and vertical features may be used sequentially, in parallel or simultaneously. Using such non-OCR-based features provides a fast implementation and robustness in noisy environments.

  The image content base extraction unit 414 is software that extracts content base information from an image. The image content based extractor 414 performs an image comparison between the captured images and the images in the database 320 to generate a list of possible image matches and associated confidence levels. Further, each of the matching images may have associated actions and associated data that are performed in response to user input. In one example, the image content-based extraction unit 414 extracts content based on a raster image (e.g., a map) by, for example, converting the image into a vector representation, and the vector representation is the same image in the feature array. Can be used to query the image database for An alternative embodiment uses the image of color content or the geometry of objects in the image to look for matching images in the database.

  A steganographic correction unit 416 is a software application that executes a steganographic correction prior to printing. To better enable the MMR application, digital information is added to the text and image before the text and image are printed. In an alternative embodiment, the steganographic modifier 416 generates and stores an MMR document, which is 1) original based content such as text, audio or video information, 2) text, audio, video, applet, hyper Includes some form of additional content such as text links. Steganographic modifications include encoding digital information by embedding watermarks in color or grayscale images, printing dot patterns in the background of documents, and slight modifications to the contours of printed characters.

  The paper reading history log 418 is a paper document reading history log. The paper reading history log 418 is, for example, in the document event database 320. The paper reading history log 418 may be based on a document identification-from-video technology developed by Ricoh Innovations. The paper reading history log 418 is particularly useful, for example, when reading a document to the MMR user 110 and / or reminding any related event.

  The online reading history log 420 is an online document reading history log. Online reading history log 420 is in document event database 320 based on analysis of operating system events. The online reading history log 420 is a record of an online document read by the MMR user 110, and a part of the document is read. Entries in the online reading history log 420 may be printed in any number of subsequent printouts in a number of ways, for example, providing notes under each page based on the time spent reading each part. Or by highlighting text in different colors. Further, the multimedia annotation software 410 may indicate this data with the PD index 322. Alternatively, the online reading history log 420 may be supported by the MMR computer 112 included in a device such as a face detection system that monitors the MMR computer 112.

  The collaborative document checker 422 is a software application in which one or more readers of different versions of the same paper document can comment on the comments applied by other readers (the his / her capture device 106 can be Can be confirmed by instructing at the location). For example, the annotation may be displayed on the capture device 106 as an overlay on top of the document thumbnail. The collaborative document confirmation unit 422 may be implemented or cooperate with any existing type of collaborative software.

  The real-time notification unit 424 is a software application that performs real-time notification of a document to be read. For example, MMR user 110 reads a document, while his / her reading trace is posted on a blog or on an online bulletin board. As a result, other people interested in the same topic may gather and chat about the document.

  The multimedia acquisition unit 426 is a software application that extracts multimedia from alternative paper documents. For example, the MMR user 110 retrieves all conversations that take place, while pointing the capture device 106 at that document location, an alternative paper document is presented to the MMR user 110 desk. This assumes that an office portal 120 (or other suitable means) exists in the office of the MMR user 110 that captures the multimedia data.

  The desktop video reminder unit 428 is a software application that notifies the MMR user 110 of events that occur in the MMR computer 112. For example, by directing capture device 106 in a section of a paper document, MMR user 110 may view a video clip that shows changes in the desktop of MMR computer 112 when that section appears. Further, the desktop video reminder unit 428 may be used to retrieve other multimedia recorded by the MMR computer 112 (eg, sound near the MMR computer 112).

  Web page reminder unit 430 is a software application that reminds MMR user 110 of a web page visible on his / her MMR computer 112. For example, by panning the capture device 106 on a paper document, the MMR user 110 sees a trace of the displayed web page, while the relevant section of the document is shown on the desktop of the MMR computer 112. The web page may be shown in a browser such as the SD browser 312, 314, or on the display 212 of the capture device 106. Alternatively, the web page may be displayed on the display 212 of the capture device 106 or on the MMR computer 112 as the URL itself.

  The physical history log 432 is in the document event database 320, for example. The physical history log 432 is a physical history log of paper documents. For example, the MMR user 110 points his / her capture device 106 at a paper document and uses the information stored in the physical history log 432 to create a document of interest at some past time. Other adjacent documents are determined. This process is supported by a tracking system such as RFID. In that case, the capture device 106 includes an RFID reader 244.

  The completed document confirmation unit 434 is a software application that extracts information acquired in the past used to fill in a certain form (standard form). For example, the MMR user 110 points to his / her capture device 106 on a blank form (eg, a billing form printed from a website) and prepares a history of previously entered information. Thereafter, the completed document confirmation unit 434 automatically fills in the form with information input in the past.

  Propagation time 436 is a software application that retrieves source files in preparation for past and future versions of a document and obtains and displays a list of events associated with those versions. This process makes up for the fact that on-hand printed documents have been generated from versions of the most important external events (e.g. discussions and meetings) that have been associated with documents that were created in months.

  The location confirmation unit 438 is a software application that manages location-aware documents. The management of the position recognition document is supported by an RFID tracking system, for example. For example, the capture device 106 captures a trace of the geographic location of the MMR user 110 during the day and scans the RFID tag attached to the document or folder containing the document. The RFID scanning process is executed by the RFID reader 244 of the capturing device 106 and detects any RFID within the range. The geographical location of the MMR user 110 may be tracked by the identification number of each cell tower in the cellular infrastructure 132, by the GPS device 242 of the capture device 106, or in combination with the geolocation means 142. Alternatively, document verification may be accomplished with the video camera 232 of the capture device 106 or “always-on-video”. Location data provides a “geo-referenced” document and allows a map-based interface that shows where the document is located throughout the day. One application is a lawyer carrying a file that visits a remote client. In the alternative, the documents 118 include detection means attached to them, which can detect when the document is moved and perform some basic face detection process. This sensing function may be done via a gyroscope or similar set of devices attached to the document. Based on the location information, the MMR system 100b informs him / her that the document is moving when “calling” the owner's cellular phone. The cellular phone adds the document to a virtual briefcase. In addition, this is the “invisible” bar code concept, which is a machine readable marking that is visible to the video camera 232 or still camera 234 of the capture device 106 but is invisible or very subtle to humans. is there. Various inks and steganography or printed image watermarking techniques (which may be decoded by the capture device 106) may be considered to confirm the position.

  The PC authoring unit 440 is a software application that executes authoring processing on a PC such as the MMR computer 112. The PC authoring unit 440 is provided as a plug-in for existing authoring applications such as Microsoft Word, PowerPoint and Web page authoring packages. The PC authoring unit 440 allows the MMR user 110 to prepare a paper document that is linked to an event from his / her MMR computer 112 or to an event in his / her environment. including. Alternatively, the PC authoring unit 440 can automatically generate a paper document having a link so that the document to be printed is automatically linked to the word file from which the document is generated. Alternatively, the PC authoring unit 440 allows the MMR user 110 to search for a word file and give it to someone else. Paper documents containing links are referred to as MMR documents. Further details of the MMR document are further described in connection with FIG.

  The document authoring unit 442 is a software application that executes an authoring process for an existing document. The document authoring unit 442 can be realized as, for example, a personal edition or a business edition. For the personal edition, the MMR user 110 scans the documents and adds them to the MMR document database (eg, adds to the document event database 320). In the case of the commercial edition, the issuer (or a third party) creates an MMR document from the original electronic source (or electronic galley print). This functionality may be incorporated into a high-end publishing package (eg, Adobe leader) and linked with back-end users provided by other entities.

  The capture device authoring unit 444 is a software application that directly executes an authoring process in the capture device 106. Using the capture device authoring unit 444, the MMR user 110 takes the keyphrase from the paper document he / she has in his hand and stores the keyphrase along with the additional content captured on the spot, Create a temporary MMR document. Additionally, using the capture device authoring unit 440, the MMR user 110 returns to his / her MMR computer 112 and downloads his / her temporary MMR document to an existing document application (eg, PowerPoint). However, it may be edited into a final version of the MMR document or other standard format document for other applications. In doing so, images and text are automatically inserted into the pages of existing documents (eg, inserted into pages of PowerPoint documents).

  The unconscious upload unit 446 is a software application that uploads a print document to the capturing device 106 unconsciously (automatically and without user intervention). Since the MMR user 110 is owned by the MMR user 110 at most times, including when the MMR user 110 is at his / her MMR computer 112, in addition to sending the document to the printer 116, the printer The driver 316 couples these same documents to the storage device 216 of the capture device 106, via the wireless communication link 218 of the capture device 106, with the Wi-Fi technology 134 or the Bluetooth technology 136, or the capture device 106 couples to the MMR computer 112. If it is, it may be sent out by wired connection. In that case, the MMR user 110 will never forget to take the document after it has been printed. This is because the document is automatically uploaded to the capture device 106.

  The document version acquisition unit 448 is a software application that acquires past and future versions of a given source file 310. For example, the MMR user 110 points the capture device 106 to the location of the printed document, and the document version acquisition unit 448 identifies the current source file 310 (eg, a word file), other past and future versions of the source file 310. To do. In certain embodiments, this operation utilizes Windows file tracking software to track where the source file 310 was replicated and moved. Other such file tracking software can also be used. For example, Google Desktop Search or Microsoft Windows Search Comparison can find the current version of a file with a query consisting of words selected from the source file 310.

  The PC document metadata unit 450 is a software application that acquires document metadata. For example, the MMR user 110 points the capture device 106 to the printed document, and the PC document metadata section 450 determines who printed the document, when the document was printed, and where the document was printed. Also check the file path of the given source file at the time of printing.

  The capture device user interface (UI) unit 452 is a software application that manages the operation of the user interface (UI) of the capture device 106 and allows the MMR user 110 to interact with the paper document. The capture device user interface (UI) portion 452 and the capture device user interface (UI) 224 combine to read data from and write data to an existing document, and the enhanced reality associated with that document. Viewing and interacting with (ie, via the capture device 106, the MMR user 110 knows what happened, when the document was created or when it was edited), and he / Allows MMR user 110 to view and interact with the enhanced reality associated with the documents displayed on her capture device 106.

  The domain specific component 454 is a software application that manages domain specific functions. For example, in the case of a music application, the domain specific component 454 is a software application that, for example, matches music detected via the voice recorder 236 of the capture device 106 with a title, artist, or composer. In this way, items of interest such as music CDs and sheet music relating to the detected music are presented to the MMR user 110. Similarly, domain specific component 454 is adapted to operate in a similar manner for video content, video games, and some entertainment information. Device specific component 454 may be adapted to an electronic version of some high volume media content.

  With further reference to FIGS. 3 and 4, the software components of the MMR software suite 222 are fully contained within the MMR system 100b of one or more MMR computers 112, the network media server 114, the service provider server 122, and the capture device 106. It should be noted that may be present in part or in part. In other words, the processing of the MMR system 100b as executed by the MMR software group 222 is performed in some user-defined configuration among the MMR computer 112, the network media server 114, the service provider server 122, and the acquisition device 106. (Or may be distributed among such other processing environments included in the system 100b).

It will be apparent from this disclosure that the basic functions of the MMR system 100a / 100b can be performed with certain combinations of the software components of the MMR software suite 222. For example, basic functions according to an example of the MMR system 100a / 100b include the following:
Create and add an MMR document that includes the first media part and the second media part.

  Accessing information in the second media part using the first media part (eg paper document) of the MMR document.

  Initiate (trigger) or initiate processing in the electronic domain using the first media portion (eg, paper document) of the MMR document.

  Create or add a second media portion using the first media portion (eg, paper document) of the MMR document.

  Create or add the first media part using the second media part of the MMR document.

  Use the second media part of the MMR document to trigger or initiate processing in the electronic domain or to associate with the first media part.

MMR Document FIG. 5 shows an MMR document 500 according to one embodiment of the present invention. More specifically, the MMR document 500 of FIG. 5 includes a representation 502 of a portion of a printed document 118, an action or second media 504, an index or hot spot 506, and an electronic representation 508 of the entire document 118. The MMR document 500 is typically stored in the document event database 320, but may be stored on the capture device or on some other device coupled to the network 128. In one embodiment, a plurality of MMR documents 500 may correspond to a single printed document. In another embodiment, the structure shown in FIG. 5 may be duplicated to create multiple hot spots 506 in a single printed document. In one particular embodiment, the MMR document 500 includes a representation 502 and hotspots 506 for the page and a location within the page, and the second media 504 and the electronic representation 508 are optional and drawn as shown by the wavy lines. ing. Note that second media 504 and electronic representation 508 may be added later after the MMR document is created, if desired. This basic embodiment may be used to find a document associated with the representation or to find a specific location in the document.

  The representation 502 of the portion of the printed document 118 can be any format (image, vector, pixel, character, code, etc.) that can be used to match patterns and identify at least one location in the document. The representation 502 preferably uniquely identifies a location in the printed document. In one embodiment, representation 502 is a text fingerprint as shown in FIG. The text fingerprint is automatically captured via the PD capture module 318 and stored in the PD index 322 during the printing process. Alternatively, the text fingerprint 502 may be automatically captured via the document fingerprint verification module 226 'of the document scanner 127 and stored in the PD index 322 during the scanning process. Alternatively, the representation 502 can be an entire document, a portion of text, a word (if that word is a unique instance in the document), a portion of an image, a unique attribute, or any other collable portion of the document. Any expression is acceptable.

  The action or second media 504 is preferably a digital file or some type of data structure. The second media 504 in most basic embodiments may be text that is presented or one or more commands that are executed. The second media type 504 is more generally a text file, audio file, or video file, which is associated with a portion of the document identified by the representation 502. The second media type 504 may be a data structure or file that refers to or includes a plurality of different media types or a plurality of files of the same type. For example, the second media 504 can be text, commands, images, PDF files, video files, audio files, application files (eg, spreadsheets or word processing documents), etc.

  The index or hot spot 506 is a link between the representation 502 and the action or platform 2 media 04. Hot spot 506 associates representation 502 and second media 504. In one embodiment, the index or hot spot 506 includes positional information such as x and y coordinates in the document. The hot spot 506 may be a certain point (point), a certain area, or even the entire document. In one embodiment, the hot spot is a data structure with a pointer to the representation 502, a pointer to the second media 504, and a location in the document. The MMR document 500 can include multiple hot spots 506, in which case the data structure creates links between multiple representations, multiple second media files, and multiple locations within the printed document 118. Should be understood.

  In an alternative embodiment, the MMR document 500 includes an electronic representation 508 of the entire document 118. This electronic representation can be used to locate the hot spot 506 and display the document on the capture device 106 or MMR computer 112 using a user interface.

  Examples of using the MMR document 500 are as follows. By analyzing the text fingerprint or representation 502, the captured text fragments are verified by the document fingerprint verification module 226 'of the capture device 106. For example, the MMR user 110 points the video camera 232 or still camera 234 of his / her capture device 106 to the print document 118 and captures the image. Next, the document fingerprint matching module 226 performs an analysis on the captured image to see if there is an associated entry in the PD index 322. If there is a match, the presence of the hot spot 506 is highlighted to the MMR user 110 on the display 212 of his / her capture device 106. For example, words or phrases as shown in FIG. 5 are emphasized. Each hot spot 506 in the printed document 118 serves as a link to other user determined or predetermined data (eg, one of the MM files 336 on the network media server 114). Accessing the text fingerprint or representation 502 stored in the PD index 322 allows electronic data to be appended to any MMR document 500 or any hot spot 506 in the document. As described with reference to FIG. 4, a paper document that includes at least one hot spot 506 (eg, a link) is referred to as an MMR document 500.

  Examples of operation of the MMR system 100b with respect to FIGS. An MMR user 110 or some other entity (eg, a publishing company) opens a given source file 310, initiates the printing process, and creates a paper document, such as a printed document 118. During the printing process, the following operations (1) to (6) are automatically executed. (1) Automatically capture the print format via the PD capture module 318 during printing and transfer it to the capture device 106. The electronic representation 508 of the document is automatically captured at the time of printing, for example, using the PD capture module 318 at the output of the SD browser 312. For example, the MMR user 110 prints content from the browser 312, and the content is selected (filtered) by the PD capture module 318. As described above, the two-dimensional arrangement of characters on the page can be determined when the document is laid out for printing. (2) The document source file 310 is automatically captured by the PD capture module 318 during printing. (3) Analyzing the printed format and / or source file 310 via the document analysis module 326 to obtain “nominated entities” or other information of interest input to the multimedia annotation interface of the capture device 106 To discover. The named entity is, for example, an “anchor” for later granting multimedia, and a hot spot 506 is automatically generated. Document analysis module 326 receives an input source file 310 associated with a given print document 118. The document analysis module 326 is an application that confirms the representation 502 (eg, title, author, time or location) used with the hotspot 506 in the document 118 and prompts the capture device 106 to receive the information. (4) Automatically index printed format and / or source file 310 in preparation for content-based search (build PD index 322). (5) Create an entry in the document event database 320 for the document and create an event associated with the source file 310, such as history and current location. (6) Perform interactive dialogue within the printer driver 316. This allows the MMR user 110 to attach a hot spot 506 to the document before the document is printed and an MMR document 500 is created. Related data is stored in the MMR computer 112 or uploaded to the network media server 114.

Alternative embodiments
The MMR system 100 (100a or 100b) is not limited to the configurations shown in FIGS. 1A-1B, 2A-2D, and 3-5. The MMR software may be distributed throughout or in part between the capture device 106 and the MMR computer 112 and requires significantly less than all the modules described with respect to FIGS. Possible configurations include the following alternatives:

  The MMR system 100 according to the first alternative includes a capture device 106 and capture device software. The capture device software is a capture device user interface (UI) 224 and a document fingerprint verification module 226 (eg, as shown in FIG. 3). The capture device software may be executed on the capture device 106 or may be executed on an external server such as a network media server 114 or service provider server 122 accessible to the capture device 106. In this alternative, a network service that provides data linked to the publication is available. A hierarchy determination method may be used. In that method, a publication is first verified and the pages and sections within the publication are verified.

  The MMR system 100 according to the second alternative includes a capture device 106, capture device software, and document usage software. A second alternative includes software such as that shown and described with reference to FIG. 4 to capture and index printed documents and tie basic document events such as document editing history. This allows the MMR user 110 to point his / her capture device 106 to some printed document and to see the name and location of the source file that generated the document, as well as the time and location of the printing. .

  The MMR system 100 according to a third alternative includes a capture device 106, capture device software, document usage software, and an event capture module 324. An event capture module 324 is added to the MMR computer 112 that captures the event, which is when the document becomes visible on the desktop of the MMR computer 112 (as confirmed by monitoring the GDI character generator), An event related to a document, such as a URL accessed when the document is opened, or a character typed on the keyboard when the document is opened.

  The MMR system 100 according to a fourth alternative includes a capture device 106, capture device software and a printer 116. In a fourth alternative, printer 116 is equipped with a Bluetooth transceiver or similar communication link to communicate with capture device 106 of any MMR user 110 in close proximity. Whenever any MMR user 110 retrieves a document from the printer 116, the printer 116 pushes the MMR data (document layout and multimedia clips) to the user's capture device 106. User printer 116 includes a keypad that allows a user to log in and enter a code to obtain multimedia data associated with a particular document. A printed representation of the code inserted by the printer driver 316 may be included in the footer of the document.

  An MMR system 100 according to a fifth alternative includes a capture device 106, capture device software and an office portal 120. The office portal 120 is preferably a personalized office portal 120. The office portal 120 captures events in the office such as conversations, conferences / phones, meetings and the like. The office portal 120 identifies and tracks specific paper documents on the physical desktop. In addition, the office portal 120 executes document verification software (ie, the document fingerprint verification module 226 and the host document event database 320). A fifth alternative functions to distribute the computational burden from the MMR computer 112 and provides a convenient way to package an MMR system 100b, such as a customer device (eg, the MMR system 100b is Apple Computer, Inc.'s Sold as hardware and software products running on Mac minicomputers.)

  The MMR system 100 according to the sixth alternative includes a capture device 106, capture device software and a network media server 114. In this alternative, the multimedia data resides on a network media server 114, such as a camcast video on demand server. If the MMR user 110 scans a portion of the document text by the user of his / her capture device 106, the resulting lookup command will be sent to the set top box 126 (over the air, over the Internet, or over the telephone line). Called) is communicated to the set-top box 126 associated with the cable TV of the MMR user 110 or to the Comcast server. In either case, the multimedia is streamed from the Comcast server to the set top box 126. System 100 knows where to send data. This is because MMR user 110 has registered his / her phone call in the past. Accordingly, the capture device 106 can be used to access and control the set top box 126.

  An MMR system 100 according to a seventh alternative includes a capture device 106, capture device software, a network media server 114, and a location service. In this alternative, the location awareness service identifies multiple destinations for output from the Comcast system (or other suitable communication system). This function is performed by automatically identifying the cellular telephone tower ID or by a keypad interface that allows the MMR user 110 to select the location where the data is displayed. Thus, as long as the other location has cable access, the user can access cable TV screenings and programs provided by the cable operator when visiting the other location.

Document fingerprint verification (image-based patch determination)
As described above, document fingerprint verification involves uniquely identifying a portion of an MMR document, a “patch”. Referring to FIG. 6, a document fingerprint verification module / system 610 receives a captured image 612. The document fingerprint matching system 610 queries a set of pages in the document database 3400 (eg, described below with reference to FIG. 34) and returns a list of pages and documents that contain the captured image 612. Each of the results is an xy location where the captured input image 612 appears. Those skilled in the art will appreciate that the database 3400 is outside the document fingerprint matching module 610 (eg, as shown in FIG. 6), but may be within the document fingerprint matching module (eg, As shown in FIGS. 7, 11, 12, 14, 20, 24, 26, 28, and 30-32, the document fingerprint verification module 610 includes a database 3400).

  FIG. 7 shows a block diagram of a document fingerprint matching system 610 according to one embodiment of the present invention. The capture device 106 captures an image. The captured image is transmitted to the quality assessment module 712 to effectively make a preliminary determination on the content of the captured image based on the need and capability of downstream processing. For example, if the captured image is of a quality that cannot be processed downstream by the document fingerprint verification system 610, the quality assessment module 712 causes the capture device 106 to reacquire the image at a higher resolution. . In addition, the quality assessment module 712 may detect many other related attributes of the captured image--for example, the sharpness of text contained in the captured image (the sharpness may be captured). It is an indicator of whether the image is aimed.) Furthermore, the quality assessment module 712 may check whether the captured image contains something that can be part of the document. For example, an image patch that includes something that is not a document image (eg, desk, outside scene) indicates that the user is moving the view of the capture device 106 to a new document.

  Further, in one or more embodiments, the quality assessment module 712 performs text / non-text determination and passes only the image to include recognizable text. FIG. 8 illustrates a flowchart for distinguishing text / non-text according to one or more embodiments. In step 810, the number of columns of pixels is extracted from the input image patch. In general, the input image is grayscale and each value in the column is an integer from zero to 255 (for 8-bit pixels). In step 812, local pixels in each column are detected. This can be done by the generally understood “sliding window” method, in which a fixed-length window (eg, N pixels) is slid M pixels at a time along the column (where M <N.) At each step, the presence or absence of a peak is determined by looking for significant differences in gray level values (eg, greater than 40). When a peak is identified at a location in the window, detection of other peaks is refrained whenever the sliding window overlaps the location. In step 812, a gap between a series of peaks is also detected. Step 812 is applied to column number C in the image patch, and in step 814 the gap value is accumulated in the histogram.

  The gap histogram is compared with other histograms derived from training data along with known classifications stored in database 818 (step 816), and a decision regarding the category of the patch (whether it is text) Output with reliability index. The histogram classification at step 816 takes into account the general appearance of the histogram derived from the image of the text, which is based on the distance between lines along with two close peaks and one or two other sufficiently small peaks. Includes concentrated ones (integer times higher in histograms away from those peaks). Classification may determine the shape of the histogram along with an indicator of statistical variation, or the histogram may be compared with a stored prototype one by one with a distance indicator (for example, the distance indicator may be a Hamming distance or Euclidean distance). .)

  Referring to FIG. 9, an example of text / non-text determination is shown. The input image 910 is processed to sample the number of columns, a subset of which is shown as a wavy line. An exemplary column 912 grayscale level is shown at 914. The Y value is the grayscale level at 910 and the X value is the 910 row. The gap detected between the peaks in the histogram is shown at 916. A histogram of gap values from all sampled columns is shown at 918. This example shows the shape of a histogram derived from a patch containing text.

  FIG. 10 shows a flowchart for estimating the point size of the text in the image patch. This flowchart takes into account that the blur of the image is inversely proportional to the distance of the capture device from the page. By estimating the blur, the distance may be estimated, and the distance may be used to scale the size of the object in the image to a known “normalized” size. This behavior may be used to estimate the focus size of the text in the new image.

  In the training phase 1010, an image of a patch of text of known font and point size (referred to as a “proof” image) is acquired by the image capture device at a known distance in step 1012. The size (expressed in pixels) of the text character in the image is measured at step 1014. This may be done manually with an image annotation tool such as a Microsoft photo editor. The blur of the calibration image is estimated at step 1016. This may be performed, for example, with a known measure of the spectral cut-off of the two-dimensional fast Fourier transform. This may be expressed in units of a large number of pixels.

When a “new” image is provided to the MMR recognition system at run-time in step 1024, the image is processed in step 1026 to identify the text using commonly known methods of line splitting and character splitting. A box surrounding each character is generated. The height of these boxes may be expressed by the number of pixels. The blur of the new image is estimated at step 1028 in a manner similar to that at step 1016. These estimates are combined at step 1030 to produce a first estimate 1032 of the point size for each character (or equivalently for each line). This may be done by calculating the following formula:
(Proof image unclear size / new image unclear size) ×
(New image text height / Structure image text height) x
(Proofreading image font size points)
This scales the point size of the text in the proof image and generates an estimated point size of the text in the input image patch. The same scaling function may be applied to the height of all of the character's siege boxes. This results in a determination for all the characters in the patch. For example, if the patch contained 50 characters, this would yield 50 votes for the point size of the font in the patch. One estimate for the point size may be derived along with the median (median) of the vote.

  Furthermore, referring more specifically to FIG. 7, in one or more embodiments, feedback of the quality assessment module 712 to the capture device 106 may be directed to a user interface (UI) of the capture device 106. . For example, the feedback includes an instruction in the form of a sound or vibration that includes something that the captured image looks like text, but it is smeared and the user installs the capture device 106 You may indicate that it should. The feedback includes a command that may change the optical parameters of the capture device 106 to improve the quality of the captured image. For example, the focus, F-stop, and / or exposure time may be adjusted to improve the quality of the captured image.

  Further, the feedback of the quality assessment module 712 to the capture device 106 may be customized depending on the needs of the feature extraction algorithm used. As described below, feature extraction converts an image into a symbolic representation. In recognition systems that calculate word length, it may be desirable to blur the captured image for the optical properties of the capture device 106. Those skilled in the art will note that such an adjustment forms an image but is probably not recognized by the human or optical character recognition (OCR) process and is well suited to feature extraction techniques. The quality assessment module 712 does this by feeding back instructions to the capture device 106, which blurs the focus of the lens and generates a blurred image.

  The feedback process is modified by the control structure 714. In general, control structure 714 receives data and symbolic information from other components in document fingerprint matching system 610. The control structure 714 can determine the order of execution of various procedures in the document fingerprint matching system 610 and optimize the computational load. The control structure 714 identifies the xy coordinates of the received image patch. More specifically, the control structure 714 can receive information about the need for the feature extraction process, the results from the quality assessment module 712, and the parameters of the capture device 106 and change them as appropriate. it can. This can be done dynamically in a frame-by-frame format. In system configurations that use multiple feature extraction methods, one may require a blurred image of a large patch of text and another may require a high resolution with a sharp focus on the paper eye. Absent. In that case, the control structure 714 sends a command to the quality assessment module 712 to direct the appropriate image quality if the view contains text. The quality assessment module 712 interacts with the capture device 106 to produce the correct image (e.g., M images that are sharply focused (high resolution) in the paper eye, followed by N blurred images). Followed by large patches.) The control structure 714 tracks the progress of those image processing in the pipeline process and ensures that the corresponding feature extraction and classification is applied.

  Image processing module 716 modifies the quality of the input image based on the needs of the recognition system. Specific examples of the type of image correction are sharpening, deskewing and binarization. Such an algorithm includes a number of adjustable parameters such as mask size, predicted rotation amount and threshold.

  As shown in FIG. 7, the document fingerprint matching system 610 dynamically modifies the parameters of the image processing module 716 using feedback from the feature extraction and classification modules 718, 720 (described below). This works because users typically point their capture device at the same location on the document for several seconds in a row. For example, if the capture device 106 processes 30 frames per second, the processing results of the first few frames in any sequence will affect how much later captured frames are processed.

  The feature extraction module 718 converts the captured image into a symbolic representation. In one embodiment, the feature extraction module 718 identifies words and calculates a box around them. In another embodiment, feature extraction module 718 identifies connected components and provides a description of their shape. Further, in one or more embodiments, the document fingerprint matching system 610 shares metadata about the feature extraction results with the control structure 714 and uses the metadata to adjust parameters of other system components. One skilled in the art will recognize that this significantly reduces computational constraints and significantly improves accuracy by preventing the recognition of poor quality data. For example, the feature extraction module 718 that identifies a word enclosing box notifies the control structure 714 of the number of lines and the found “word”. If there are too many words (eg, indicating that the input image is to be split), the control structure 714 may instruct the quality assessment module 712 to create a blurred image. The quality assessment module 712 then sends an appropriate signal to the capture device 106. Alternatively, the control structure 714 may instruct the image processing module 716 to apply a smoothing filter.

  The classification module 720 converts the feature description from the feature extraction module 718 into the identity of one or more pages in the document and the x and y coordinates (where the input image patch appears) in those pages. Its identity is formed depending on the feedback from the database 3400 as described above. Further, in one or more embodiments, a confidence value may be associated with each determination. Document fingerprint verification system 610 may utilize such determination to determine parameters of other components in the system. For example, if the reliability of the top two determinations are close to each other, the control structure 714 may determine that the parameters of the image processing algorithm should be changed. This increases the extent of the median filter range and carries the resulting downstream to the remaining components.

  Furthermore, there may be feedback between the classification module 720 and the database 3400, as shown in FIG. Further, those skilled in the art will recall that the database 3400 may be external to the module 610 as shown in FIG. Verification of patch identity can be used to query the database 3400 for other patches that have a similar appearance. This is not a comparison between the input image patch and the database 3400, but the complete image data of the patch stored in the database 3400 is compared with other images in the database 3400. This provides an additional level of confirmation regarding the determination of the classification module 720 and allows some preliminary processing of the matching data.

  The database comparison can also be executed in symbolic expression for patches other than those of only image data. For example, it is assumed that the best determination content is that the image patch is every other line in a 12-point Arial font. Database comparisons may identify patches in other documents with similar fonts, line spacing, and word layouts using only text metadata other than image data comparisons.

  Database 3400 may support several types of content-based queries. The classification module 720 may communicate a feature arrangement to the database 3400 and receive a list of xy coordinates and documents in which the arrangement appears. For example, the feature may be a trigram (described later) having a word length vertically and horizontally. The database 3400 may be organized to return a list of results depending on any type of query. The classification module 720 or the control structure 714 may combine their rankings to generate a single sorted decision list.

  There may be feedback between the database 3400, the classification module 720 and the control structure 714. In addition to storing sufficient information to locate from feature vectors, the database 3400 may store related information including the original image of the document as well as symbolic representations of the graphical components. This allows the control structure 714 to control the operation of other system components in place. For example, if there are two plausible decisions for a given image patch, the database 3400 may instruct them to clarify them by reducing with respect to the display of the image and examining the right side of the region. The control structure 714 sends an appropriate message to the capture device 106 instructing to reduce. The feature extraction module 718 and the classification module 720 examine the right side of the image for the image printed on the document.

  Furthermore, it should be noted that the database 3400 stores detailed information about the data surrounding the image patch if the patch is properly located in the document. This may be used to trigger additional hardware and software image analysis steps not previously anticipated. The detailed information is prepared, for example, by a print capture system that stores a detailed symbolic description of the document. In one or more other embodiments, similar information may be obtained by scanning a document.

  Still referring to FIG. 7, the position tracking module 724 receives information about the identity of the image patch from the control structure 714. The position tracking module 724 uses it to retrieve a copy of the entire document page from the database 3400 or retrieve a data structure describing the document. The first place for the start of the position tracking process is the anchor. The position tracking module 724 receives image data from the capture device 106 when the quality assessment module 712 confirms that the captured image is suitable for tracking. The position tracking module 724 also has information on the time that has elapsed since the last frame was successfully recognized. The position tracking module 724 uses a selective flow processing technique that allows a distance to be estimated on the document if the capture device 106 is moved between successive frames. Based on a given sampling rate of the capture device, the search object can be estimated even if the data seen by the device is not recognizable. The estimated position of the capture device 106 may be confirmed by comparing the image data with related image data derived from a database document. A simple example is to calculate the cross-correlation between the captured image and the expected image in the database 3400.

  Thus, the position tracking module 724 provides interactive usage of the database image and guides the progress of the position tracking algorithm. This allows attachment for electronic interaction with non-text objects such as graphics and images. Furthermore, in one or more other embodiments, such attachments may be implemented without performing the image comparison / confirmation described above. In other words, by estimating the instantaneous movement of the capture device 106 with respect to the page, an electronic link in the field of view not related to the captured image may be estimated.

  FIG. 11 illustrates a document fingerprint verification method according to another embodiment of the present invention. The “feed forward” method shown in FIG. 11 processes each patch independently. This method retrieves from an image patch the features used to identify one or more pages and the xy coordinates (where the patch appears) on that page. For example, in one or more embodiments, feature extraction for document fingerprint matching may depend on horizontal and vertical groupings of captured image features (e.g., words, characters, blocks, etc.). Also good. The group of extracted features may be used to look for documents (and patches in the document) that contain the extracted features. The OCR function may be used to identify horizontal word pairs in the captured image. Each identified horizontal word pair is used to form a search query against the database 3400, which searches all documents including the identified horizontal word pair and the x− of the word pairs in the document. This is for confirming the y coordinate. For example, if the horizontal word pair is “that, cat” (“the, cat”), the database 3400 returns (15, x, y), (20, x, y) and the horizontal word pair “ Indicates that “cat” appears in the xy coordinates of documents 15 and 20. Similarly, for each vertical word pair, the database 3400 is queried to find all documents that contain that word pair instance and the xy coordinates of the word pairs in the document. For example, for the word pair “in, hat” (“in, hat”) adjacent in the vertical direction (vertical direction), the database 3400 returns (15, x, y), (7, x, y) Indicates that the word pair “in, hat” adjacent to appears in the xy coordinates of document 15,7. Thus, using the document and location information returned from the database 3400, a determination is made as to which document represented the greatest location overlap between the various horizontal and vertical word pairs retrieved from the captured image. be able to. This will confirm the document containing the captured image in response to the confirmation of the presence of the hotspot and linked media.

  FIG. 12 illustrates a document fingerprint verification method according to another embodiment of the present invention. The “interactive image analysis” method shown in FIG. 12 involves an interaction between image processing and feature extraction and may be done before the image patch is recognized. For example, the image processing module 716 may first estimate the unclearness of the input image. Then, the feature extraction module 718 calculates the distance from the page of the image text and the point size. Then, the image processing module 716 may execute a template matching step for the image using the font character of the point size. Thereafter, the feature extraction module 718 may extract character or word features from the execution result. Further, those skilled in the art will recognize that fonts, point sizes and features are constrained by the fonts in the document database 3400.

  FIG. 13 shows an example of interactive image analysis as described above with respect to FIG. The input image patch is processed at step 1310 to estimate not only the distance from the camera but also the font and point size of the text in the image patch. Those skilled in the art will note that font estimation (i.e., identifying font candidates for text in a patch) may be done with known techniques. Point size and distance estimation may be performed, for example, using the flow process described with respect to FIG. In addition, other techniques may be used such as known focal length methods that are readily applicable to the capture device.

  Still referring to FIG. 13, a line splitting algorithm is applied at step 1312 to build a box that surrounds the line of text in the patch. The height of each line image is normalized to a fixed size at step 1314 using known methods such as proportional scaling. In addition to the font size, the identity of the font detected in the image is communicated to a collection of font prototypes, which are used to retrieve the character's image prototype for each named font.

  The font database 1322 may be constructed from a font collection in the user's system, which is used by the operating system and other software applications for printing documents (e.g., Microsoft Windows raster fonts, TrueType, OpenType, etc.) In one or more other embodiments, the font collection may be generated from an original image of a document in the database 3400. The xml file of database 3400 may be used to prepare xy coordinates surrounding the box and extract an accurate prototype image of the character from the original image. The Xml file accurately specifies the name of the font and the font size of the character.

  The character prototype in the selected font is the size normalized in step 1320 based on the function of the parameters used in step 1314. In the image classification in step 1316, the character of the standardized size output in step 1320 is compared with the output in step 1314, and a determination is made for each xy coordinate in the image patch. Existing image template matching methods are used to generate output such as (ci, xi, yi, wi, hi), where ci is the character's identity and (xi, yi) encloses the box , Hi, wi are the height and width thereof, and i = 1,..., N indicate the characters detected in the image patch.

  In step 1318, the search in the geometric constraint database is performed as described above, but in this case, it may be specialized for character pairs instead of word pairs. In that case, “ab” indicates that a and b are adjacent in the horizontal direction; “a + b” indicates that they are adjacent in the vertical direction; “a / b” is indicates that a is southwest of b; “a \ b” indicates that a is southeast of b. The geometric positional relationship may be derived from the values of xi and yi for each pair of characters. The MMR database 3400 is organized so that it returns a list of document pages that contain character pairs rather than word pairs. The output of step 1326 is a list of candidates that match the input image expressed as n sets ranked by score (documenti, pagei, xi, yi, actioni, scorei).

  FIG. 14 illustrates a document fingerprint verification method according to another embodiment of the present invention. The “generation and inspection” method shown in FIG. 14 processes each patch independently. The method extracts features from the image patch that are used to identify the number of image pages that may contain a given image patch. Further, in one or more embodiments, additional extraction and classification steps may be performed to rank each page with a likelihood that the page contains image patches.

  With further reference to the “generation and inspection” method described with reference to FIG. 14, the captured image features are extracted and the image patches in the database 3400 that contain most of the extracted features are identified. Also good. The first X document patches (candidates) with the best matching features are further processed. In this process, the relative positions of features in matching document patch candidates are compared to the relative locations of the current image features. A certain score is calculated based on the comparison. Then, the highest score corresponding to the most matching document patch P is confirmed. If the highest score is greater than the threshold applied, the document patch P is found as matching the image being queried. The threshold is adaptive for many parameters (eg including the number of extracted features). In the database 3400, it is known where the document patch P came from, and the image being queried is confirmed as coming from that location.

  FIG. 15 shows an example of a word bounding box determination algorithm. An input image patch 1510 after image processing for correcting the rotation amount is shown. As commonly known as a deskew algorithm, this type of technique rotates the text image so that the text image is aligned with the horizontal axis. The next step of the bounding box determination is the calculation of the horizontal projection profile (plan view characteristics) (1512). The threshold for line determination is selected (1516) by a known adaptive threshold method or by a sliding window algorithm, in which the “exceeding threshold” region corresponds to a line of text. The regions within each row are extracted and processed in the same manner as 1514 and 1518 to identify the regions in the row that exceed the threshold representing a word. An example of a bounding box detected in a line of text is shown at 1520.

  Various features may be retrieved for comparison with document patch candidates. For example, scale invariant feature transform (SIFT) characteristics, corner characteristics, protruding points, ascenders, descenders, word boundaries, spaces, etc. may be extracted for matching. One feature that can be easily extracted from document images is word boundaries. As word boundaries are extracted, they are formed into groups as shown in FIG. In FIG. 16, for example, vertical groups are formed such that word boundaries have overlapping word boundaries above and below, and the total number of overlapping word boundaries is at least three (minimum overlapping word boundaries). Note that the number may be different in one or more other embodiments.) For example, the first feature point (the second word box in the second row with a length of 6) has two word boundaries above (they are 5 and 7), It has one word boundary below (its length is 5). The second feature point (the fourth word box in the third row, which has a length of 5) has two word boundaries above (they are 4 and 5 in length), and the bottom Have two word boundaries (their lengths are 8 and 7). Thus, as shown in FIG. 16, the illustrated features are expressed with the length of the middle word boundary following the length of the upper word boundary and the length of the lower word boundary. Furthermore, note that the length of the word box may be based on any metric. Therefore, it is possible to prepare different lengths for some word boxes. In that case, features including all or part of those alternatives may be extracted.

  Further, in one or more embodiments, features may be extracted such that spaces are represented by 0 and word regions are represented by 1. FIG. 17 shows a specific example. The block representation on the right corresponds to the word / space area of the left document patch.

  The extracted features may be compared with various distance metrics (eg, including norm and hamming distance). Alternatively, in one or more embodiments, a hash table may be used to identify document patches that have the same characteristics as the query image (query image). Once such a patch is identified, the angle from each feature point to another feature point may be calculated as shown in FIG. Alternatively, the angle between groups of feature points may be calculated. 1802 represents angles 1803, 1804, and 1805 calculated from the three feature points. The calculated angle is compared with the angle from each feature point to another feature point in the query image. If some angle of feature points to be matched is similar, the similarity score may be increased. Alternatively, if an angle group is used, and the angle groups between similar groups of feature points in the two images are numerically similar, the similarity score may be increased. Once the score between the query image and each extracted document patch is calculated, the document patch that yields the highest score is selected and compared to an adaptive threshold to determine if the degree of match meets certain predetermined criteria. Confirm whether or not. If it meets the criteria, it is notified that a matching document patch has been found.

  Further, in one or more embodiments, the extracted features may be based on word length. Each word is divided into estimated characters based on the height and width of the word. When the upper and lower word lines of a given word are scanned, a binary value is assigned to each estimated character according to the space information in the upper and lower word lines. The binary code is expressed as an integer. For example, referring to FIG. 19, an array of word boxes is shown, each word box representing a detected word in the captured image. Word 1910 is divided into estimated characters. This feature is described with (i) the length of word 1910, (ii) the text array in the line above word 1910, and (iii) the text array in the line below word 1910. The length of word 1910 is measured by the estimated number of characters. The text arrangement information is derived from the binary code of the space information above and below the current estimated character. For word 1910, only the last estimated character is above the space, and the second and third estimated characters are below the space. Thus, the features of word 1910 are encoded as (6,100111,111110), where 0 means a space and 1 means no space. When rewritten in integer form, word 1910 is encoded as (6,39,62).

  FIG. 20 illustrates a document fingerprint verification method according to another embodiment of the present invention. The “multiple classification” method shown in FIG. 20 uses complementary information by classifying complementary information of various feature descriptions and combining the results. A specific example of this paradigm applied to text patch matching is to extract the length of horizontally and vertically adjacent word pairs and individually calculate the ranking of the patches in the database. More specifically, for example, in one or more embodiments, the location of a feature is ascertained by a “classifier” attendant along with a classification module 720. The captured image is left to fingerprint verification using a combination of the vertical and horizontal feature classifiers of the captured image. This means that an image of text contains two independent sources for its identity-in addition to the side-by-side of the word, a vertical word layout can be used to identify the document (the document is an image extractor). It is executed from the viewpoint of For example, as shown in FIG. 21, the captured image 2110 is classified by a horizontal classifier 2112 and a vertical classifier 2114. In addition to inputting captured images, each of classifiers 2112 and 2114 retrieves information from database 3400 and outputs a ranking of document pages to which each classifier has been applied. In other words, the multiple classifier method shown in FIG. 21 classifies captured images independently using horizontal and vertical features. The ranked list of document pages is combined by a combining algorithm 2118 (an example will be described below) to output a ranked list of document pages, the list of captured images 2110 Based on both horizontal and vertical features. In particular, in one or more embodiments, separate rankings from horizontal classifier 2112 and vertical classifier 2114 can provide certain information (information about how detected features appear together in database 3400). Combined.

  Referring to FIG. 22, a specific example of how a vertical layout is integrated with a horizontal layout in preparation for feature extraction is shown. In (a), a captured image 2200 is shown along with word splitting. From the captured image, horizontal and vertical “n-grams” are determined. An “n-gram” is a sequence of several n describing the amount of a certain feature. For example, a horizontal trigram specifies the number of characters in each word in a horizontal sequence of three words. For example, in the case of captured image 2200, (b) is 5-8-7 (the number of characters in each of the words “upper”, “division” and “courses” side by side in the first row of captured image 2200), 7-3-5 (number of characters in the words “Project”, “has” and “begun” side by side in the second row of the captured image 2200), 3-5-3 (second row of the captured image 2200) Next to the words “has”, “begun” and “The”, the number of characters in each row), 3-3-6 (the words “461”, “and” next to the third row of the captured image 2200) ”And“ permit ”for each character) and 3-6-8 (number of characters for each of the words“ and ”,“ permit ”and“ projects ”side by side in the third row of the captured image 2200) Indicates.

  A vertical trigram specifies the number of characters in each vertical word above and below a given word. For example, in the case of the captured image 2200, (c) is 5-7-3 (the number of characters in each of the vertically aligned words “upper”, “Project”, and “461”), 8-7-3 (the vertically aligned words) “Division”, “Project” and “461” each character count), 8-3-3 (vertical word “division”, “has” and “and” character count each), 8-3-6 ( Vertical number of words “division”, “has” and “permit” for each character), 8-5-6 (vertical word “division”, “begun” and “permit” for each character), 8- 5-8 (number of characters for vertical words “division”, “begun” and “projects”), 7-5-6 (number of characters for vertical words “courses”, “begun” and “permit”) ), 7-5-8 (the number of characters in each of the vertical words “courses”, “begun” and “projects”), 7-3-8 (the vertical words “courses”, “The” and “projects”) Number of each character), 7-3-7 (vertical word “Project Shows a vertical trigrams "461" and "student" each number of characters) and 3-3-7 (vertically arranged word "HAS", "and" and "student" number each character).

  Based on the horizontal and vertical trigrams determined from the captured image 2200 shown in FIG. 22, a list of documents (d) and (e) is generated showing the documents including horizontal and vertical trigrams, respectively. For example, in the case of (d), the horizontal trigram 7-3-5 appears in the documents 15, 22, and 134. Further, for example, in the case of (e), the vertical trigram 7-5-6 appears in the documents 15 and 17. Using the document lists (d) and (e), a ranked list of all referenced documents is shown in (f) and (g), respectively. For example, in (f), document 15 is referenced by five horizontal trigrams in (d), while document 9 is referenced by only one trigram in (d). Further, for example, in (g), document 15 is referenced by 11 vertical trigrams in (e), while document 18 is referenced by only one vertical trigram in (e). .

  Referring to FIG. 23, a method for combining horizontal and vertical trigram information described with reference to FIG. 22 is shown. This method uses information about the known physical location of the trigram on the original printed page to combine the list of votes from the horizontal and vertical feature extraction. For all documents that are common among the top M choices output by each horizontal and vertical classifier, the location of all horizontal trigrams voted for the document is compared to the location of all vertical trigrams voted for that document. The Documents that overlap any vertical trigram receive a vote count equal to the number of horizontal trigrams, where “overlap” occurs when the bounding boxes of the two trigrams overlap. In addition, the xy coordinates of the center of overlap are counted by an appropriately modified version of the evidence accumulation algorithm described below with respect to 3406 in FIG. For example, as shown in FIG. 23, the lists of (a) and (b) (respectively (f) and (g) in FIG. 22) are divided and lists of pages referenced in both horizontal and vertical trigrams. Specify (c). Using the split lists (c), lists (d) and (e) (only the split documents referenced by the specified trigram are shown), and the printed document database 3400, document overlap It is confirmed. For example, document 6 is referenced by horizontal trigram 3-5-3 and by vertical trigram 8-3-6, and these two trigrams overlap with respect to the word “has” in captured image 2200, so document 6 is Get one vote for each overlap. As shown in (f), for a particular captured image 2200, the document 15 gets the maximum number of votes and is confirmed as the document containing the captured image 2200. (x1, y1) is confirmed as the location of the input image in document 15. In the document fingerprint matching method described above with reference to FIGS. 22 and 23, in essence, the horizontal classifier uses features derived from the horizontal array of words in the text, and the vertical classifier is derived from the vertical array of words. Features are combined, and the results are combined based on feature overlap in the original document. Such feature extraction provides a mechanism for uniquely identifying a document, while leaving the horizontal features from feature extraction to the appropriate grammar and language constraints, while not leaving the vertical features to such constraints. .

  Furthermore, while the description with respect to FIGS. 22 and 23 used trigrams in particular, any n-gram may be used for one or both of horizontal and vertical feature extraction / classification. For example, in one or more embodiments, vertical and horizontal n-grams (n = 4) may be used for multi-classifier feature extraction. In one or more other embodiments, the horizontal classifier may extract features based on n-grams (n = 3) and the vertical classifier may extract features based on n-grams (n = 5). .

  Further, in one or more embodiments, the classification may be based on adjacency relationships that are not strictly vertical or horizontal. For example, the adjacent relationship of NW, SW, SW, and SE may be used for extraction / classification.

  FIG. 24 shows an example of fingerprint verification of a document according to another embodiment of the present invention. The accuracy of the document image matching system is improved by using a document image that may match the input and determining subsequent steps in image analysis (sub-images from the initial image are matched with the input image). The “database driven feedback” method shown in FIG. 24 takes this into consideration. The method includes a transformation that duplicates the noise present in the input image. This is followed by template matching analysis.

  FIG. 25 illustrates a database-driven feedback process flow according to one embodiment of the present invention. The input image patch is first processed and recognized in steps 2510 and 2512 described above (eg, using word OCR and word pair lookup, character OCR and character pair lookup, word bounding box configuration, etc.) to identify the identity of image patch 2522. Generate a large number of candidates. Each candidate in this list contains the following items (doci, pagei, xi, yi), where doci is the document identifier, pagei is the page in the document, and (xi, yi) is the page It is an xy coordinate of the center of the middle image patch.

  The initial patch extraction algorithm in step 2512 uses the distance information from the page to selectively normalize the size of the entire input image patch to a fixed size and convert it to a known spatial resolution (e.g., 100 dpi). Guarantee. The font size estimation algorithm described above may be applied to this task. Similarly, a known distance from the focus, a depth from the focus, or the like may be used. Size normalization can scale the image patch linearly based on the height of each word enclosing box.

  The initial patch extraction algorithm queries the MMR database 3400 with the identifier, looks for each document and page, and receives it along with the center of the enclosing box of patches that the MMR database generates. The range of patches to be generated depends on the standardized input patch size. In such a method, patches with the same spatial resolution and dimensions are obtained. For example, when standardized to 100 dpi, the input patch spreads by 50 pixels on each side of the center. In this case, the MMR database instructs to generate an initial patch of 100 dpi with a height and width of 100 pixels centered on the specified xy value.

  Each initial image patch (2524) returned from the MMR database is associated with the following items (doci, pagei, xi, yi, widthi, heighti, actioni), and (doci, pagei, xei, yi) And widthi and heighti are the width and height (number of pixels) of the initial patch, and actioni is a selective action and is associated with the corresponding region in the doci entry in the database. The initial patch extraction algorithm outputs the image patch list and data, along with the configured standardized input patch size (2518).

  Further, in one or more embodiments, the patch matching algorithm (2516) compares the standardized size input patch with each of the initial patches and assigns a score that measures how well they match each other ( 2520). One skilled in the art will appreciate that a simple cross-correlation with respect to the Hamming distance is often sufficient due to a mechanism that ensures that the patch sizes are comparable to each other. In addition, the process includes introducing noise into the initial patch to simulate image noise detected at the input. The comparison may alternatively be complicated and may include a comparison of any feature group, which includes two patterns of OCR results and rankings based on the number of characters, character pairs, or word pairs, which pairs are Can be constructed from the above-described geometric positional relationship. However, in this case, the number of geometric pairs common between the input patch and the initial patch may be estimated and used as a ranking metric.

  Furthermore, the output 2520 may be in n sets (doci, pagei, xi, yi, actioni, scorei), and the score is prepared by the patch matching algorithm, and how well the input patch is in the corresponding area of doci, pagei. Measure whether they match.

  FIG. 26 shows an example of fingerprint verification of a document according to another embodiment of the present invention. The “database driven classifier” method shown in FIG. 26 uses an initial classifier to generate a set of candidates that may include an input image. These candidates are looked up in database 3400 and feature extraction plus classification methods are automatically designed for those candidates. One example is to identify an input patch that includes a Times or Arial font. In this case, the control structure 714 activates a feature extractor and classifier specialized for serif / san serif identification.

  FIG. 27 shows a flowchart of database driven classification according to one embodiment of the present invention. Following the first feature extraction 2710, the input image patch is classified 2712 by any one or more of the recognition methods described above to generate document rankings, pages and xy coordinates within the page. Each candidate in this list includes, for example, the following items (doci, pagei, xi, yi), where doci is the document identifier, pagei is the page in the document, and (xi, yi) is the page The xy coordinates of the center of the image patch inside. A patch image may be generated for each candidate using the initial patch extraction algorithm 2714 described with reference to FIG.

  Still referring to FIG. 27, a second feature extraction is applied to the initial patch (2716). This is different from the first feature extraction and may include one or more of, for example, a font detection algorithm, a character recognition technique, a bounding box and a SIEF feature. The features detected in each of the initial patches become inputs to the automatic classifier design method 2720, including, for example, neural networks, support vector machines, and / or closest classifiers (they identify unknown samples in the initial patch. Designed to be classified as one). The same second feature extraction is applied to the input image patch (2718) and the detected features become inputs to the newly designed classifier and are specific to that initial patch.

  The output 2724 may be in n sets (doci, pagei, xi, yi, actioni, scorei), and the score is prepared by a classification method (2722) automatically designed by 2720. Those skilled in the art will understand that the score measures how well the input patch matches the corresponding region of doci, pagei.

  FIG. 28 shows an example of fingerprint verification of a document according to another embodiment of the present invention. The “database driven multiple classifier” method shown in FIG. 28 reduces the possibility of unrecoverable errors early in the recognition process by using multiple candidates throughout the decision process. Some initial classification is performed. Each generates a different ranking of the input patch, which ranking can be identified by various feature extractions and classifications. For example, one in these groups is generated by n horizontal sets and is uniquely recognized by identifying the serif by sun-serif. Another example is generated by n sets in the vertical direction and is uniquely recognized from accurate calculations between rows.

  FIG. 29 shows a flowchart of database-driven multiple classification according to one embodiment of the present invention. This flow process is similar to that shown in FIG. 27 except that it uses a plurality of different feature extraction algorithms 2910, 2912 to generate independent rankings of the input image patches along with the classifications 2914, 2916. Examples of features and classification methods use n-grams of the horizontal and vertical word lengths described above. Each classifier generates a ranked list of patch identities for each candidate that includes at least the following items (doci, pagei, xi, yi, scorei), where pagei is a page in the document and (xi, yi) Is the xy coordinate of the image patch center in the page and scorei measures how well the input patch matches the corresponding location in the database document.

  The initial patch extraction algorithm described above with respect to FIG. 25 may be used to generate an initial image patch group corresponding to an entry in the list of patch identifiers at the output of 2914,2916. The third and fourth feature extractions 2918 and 2920 are applied to initial patches and classifiers that are automatically designed and applied as described in FIG.

  Still referring to FIG. 29, the rankings generated from these classifiers are combined to give one ranking with entries (doci, pagei, xi, yi, actioni, scorei) for candidate numbers i = 1, ... Generated (2924), and the value in each entry is as described above. Ranking combination 2922 may be performed, for example, using a known Borda count measure, which assigns a score to an item based on a common position in the two rankings. This is combined with the score assigned by the individual classification to produce a composite score. Furthermore, those skilled in the art will note that other methods of ranking combining may be used.

  FIG. 30 shows an example of fingerprint verification of a document according to another embodiment of the present invention. The “video sequence image storage” method shown in FIG. 30 constructs an image by integrating data from nearby or adjacent frames. An example includes “super resolution. This registers N temporarily adjacent frames and uses the point spread function of the lens, essentially sub-pixels. Perform what is edge enhancement, the effect of which increases the spatial resolution of the image, and in one or more embodiments, the super-resolution method enhances character-specific features such as holes, corners and dots. Another extension method is to specialize the super-resolution integration function using the features of candidate image patches as determined from the database 3400.

  FIG. 31 shows an example of fingerprint verification of a document according to another embodiment of the present invention. The “video sequence feature accumulation” method shown in FIG. 31 accumulates features over a number of temporarily adjacent frames before making a decision. This takes advantage of the capture device's fast sampling rate (eg, 30 frames per second) and user intent to keep the capture device directed to the same point in the document for at least a few seconds. Feature extraction is performed independently on each frame, and the results are combined to produce one integrated feature map. The binding process includes an implicit registration step. The need for this method becomes readily apparent when examining video clips of text patches. Autofocus and contrast adjustment with a typical capture device may yield significantly different results in adjacent video frames.

  FIG. 32 shows an example of fingerprint verification of a document according to another embodiment of the present invention. The “video sequence decision combining” method shown in FIG. 32 combines decisions from multiple temporarily adjacent frames. This takes advantage of the fast sampling rate and user intent of a typical capture device and directs the capture device to the same point on the document for at least a few seconds. Each frame is processed independently, generating its own ranked decision list. These decisions are combined to produce one integrated ranking of the input images. This method includes an indirect registration method that controls the decision combining process.

  In one or more embodiments, one or more of the various document fingerprint matching methods described above in connection with FIGS. 6-32 may be used in combination with one or more known matching methods (eg, Such a combination is referred to herein as “multi-tier (or multi-factor) certification”). In general, in multi-tier certification, a first matching method is used to identify a group of pages that meet certain criteria in the document database, and then a second matching method is used to identify the pages in the group. One patch is uniquely identified from the inside.

  FIG. 33 shows a flowchart of multi-tier recognition according to an embodiment of the present invention. First, at step 3310, the capture device 106 is used to capture / scan "culling" features on the document of interest. The sorting feature can be any feature, and capturing the sorting feature effectively selects a group of documents in the document database. For example, the sorting feature can be a numeric only barcode (eg, Universal Product Code (UPC)), an alphanumeric barcode (eg, Code 39, Code 93, Code 128) or a two-dimensional barcode (eg, QR Code, PDF417). , DataMatrix, Maxicode), etc. Further, the sorting feature may be, for example, a graphic, an image, a trademark, a logo, a specific color or combination of colors, a keyword, or a phrase. Further, in one or more embodiments, the sorting features may be limited to features suitable for recognition by the capture device 106.

  In step 3312, once a screening feature is captured in step 3310, a group of documents and / or pages of documents in the document database are selected in relation to the captured screening feature. For example, if the captured screening feature was a company logo, all documents in the database that are indexed to contain the logo are selected. In another example, the database may include a library of trademarks that are compared to the captured sorted images. If there is a “hit” in the library, all documents associated with the hit trademark are selected for further verification as described below. Further, in one or more embodiments, the document / page selection at step 3312 may depend on the sorting features captured on the scanned document and the location of the sorting features. For example, the information related to the captured sorting feature may specify whether the sorted image is in the upper right corner of the document rather than in the lower left corner of the document.

  In addition, confirmation that a particular captured image contains an image of the screening feature may be made by the capture device 106 or by some other component that receives raw image data from the capture device 106. Those skilled in the art will note that this is also possible. For example, the database itself may determine that a particular captured image transmitted from the capture device 106 includes a screening feature in response to the database selecting a set of documents associated with the captured screening feature. Good.

  In step 3314, after a particular group of documents is selected in step 3312, the capture device 106 continues to scan and captures an image of the document of interest. Utilizing one or more of the various document fingerprint verification methods described in connection with FIGS. 6-32, the captured image of the document is verified with the document selected in step 3312. For example, after capturing a shoe graphic image on a document of interest in step 3310, a group of documents indexed to include a shoe graphic selection feature is selected in step 3312 and A subsequent captured image of a document may be verified against a group of documents selected using the plurality of classifiers described above.

  Then, performing the multi-tier certification flow process described above in connection with FIG. 33, the patch certification time is reduced by first reducing the number of pages / documents that are subsequently matched to the captured image. In addition, the user may benefit from such improved qualification time by first scanning the document over a location (images, barcodes, graphics and other sorting features exist at that location) .) By performing such an operation, the user may quickly reduce the number of documents to be collated with images captured thereafter.

MMR Database System FIG. 34A shows a functional block diagram of an MMR database system 3400 constructed in accordance with one embodiment of the present invention. System 3400 is built for content-based extraction, where the two-dimensional geometric positional relationship between objects is represented in a searchable manner with a text-based index (or with some other searchable index). . The system 3400 uses evidence accumulation to enhance search efficiency by combining, for example, the frequency of appearance of features and the likelihood of locations within a two-dimensional region. In one particular embodiment, database system 3400 is a detailed embodiment of document event database 320 (including PD index 322), the contents of which are captured module 318 and / or document fingers described above in connection with FIG. It includes an electronic representation of the printed document generated by the print verification module 226. Other applications and configurations for the system 3400 will become apparent from the present disclosure.

  As shown, the database system 3400 includes an MMR index table module 3404 (receives a description calculated by the MMR feature extraction module 3402), an evidence storage module 3406, and a relational database 3408 (or any other suitable Storage means). The index table module 3404 queries the index table, and the index table specifies the xy coordinates, page and document of the page where each feature appears. The index table can be generated by, for example, the MMR index table module 3404 or another dedicated module. The evidence storage module 3406 is programmed or constructed to calculate a ranked group of documents, pages, and location candidates 3410 based on the data from the index table module 3404. Relational database 3408 can be used to store additional features 3412 for each patch. These include, but are not limited to, 504,508 in FIG. When deriving a signature or fingerprint (ie, a unique search term) for a patch, utilizing a two-dimensional array of text within the patch greatly increases the uniqueness of even small portions of text Increase. Other embodiments can similarly utilize any two-dimensional array of objects / features within the patch when deriving the signature or fingerprint of the patch, and embodiments of the present invention uniquely identify the patch It is not intended to be limited to a two-dimensional array of text. Other components and functions of the database system 3400 shown in FIG. 34A include a feedback-oriented feature search module 3418, a document conversion application module 3414, and a sub-image extraction module 3416. These components interact with other components of the system 3400 to perform feedback-oriented feature searches in addition to dynamically generating an initial image. Further, system 3400 includes an action processor 3413 that receives actions. The action determines the action performed by the database system 34002 and the output it provides. Each of these other components is described below.

  FIG. 34B shows an example of an MMR feature extraction module 3402 that uses a two-dimensional array of text in a patch. In one such example, the MMR feature extraction module 3402 is programmed or constructed to extract features (text or other searchable features) from the image patch utilizing OCR-based techniques. In this particular embodiment, the feature extraction module 3402 extracts the xy coordinates (location) of a word in the image of the text patch, and the location as a group of horizontally and vertically adjacent word pairs that contain it. Express. Image patches are given a “-” if they are horizontally adjacent (eg, the-cat, in-the, the-hat, is-back) and “+” if they overlap vertically. Is effectively converted into a word pair (eg, the + in, cat + the, in + is, the + back). The xy coordinate may be based on, for example, the number of pixels in each direction in the xy plane from a certain fixed point in the document image (for example, the number of pixels from the upper left corner or the center of the document). Note that while horizontally adjacent pairs may appear frequently in many other text portions in that example, vertically overlapping pairs tend not to occur frequently in other text portions. Other geometric positional relationships between image features can be encoded in the same way, eg by placing a “/” between SW-NE adjacencies and “/” between NW-SE adjacencies. Or the like. Further, the “feature” is generalized to a word enclosing box (or other feature enclosing box), and may be arbitrarily encoded unless it matches the number of characters. For example, four bounding boxes may be expressed by the character string “4rusl” as long as the upper contour has an uneven height except for a smooth lower contour. Furthermore, the geometric positional relationship may be extended to any angle and distance between features. For example, two words that are adjacent in the NW-SE positional relationship but separated by a height of 2 words (each word is described as “4ursl”) are expressed as “4rusl \\ 4rusl” Also good. Various encoding methods will be apparent from the disclosure of the present application. Furthermore, it should be noted that numbers, Boolean values, geometric shapes, and other such document features may be used for ID patches instead of word pairs.

  FIG. 34C shows an example index table organization according to one embodiment of the present invention. As shown, the MMR index table includes an inverted term index table 3422 and a document index table 3424. A unique term or feature (e.g., key 3421) points to a location in the term index table 3422 that holds the function value of the feature (e.g., key x), and the function value is stored in a list 3423 (e.g., Rec # 1 , Rec # 2, etc.), and each record identifies a candidate area of a page in the document as described above. In one example, key and key function value (keyx) are the same. In another example, a hash function is applied to key and the output of that function is keyx.

  Based on the list of query terms, all records indexed by key are inspected to identify the area that best fits all query terms. If the region has a sufficiently high match score (for example, a predetermined match threshold), it is determined as a candidate. Otherwise, the match is declared failed and no region is returned. In this embodiment, the key is a word pair (eg, “the-cat” or “cat + the”) separated by “−” or “+” as described above. This technique of incorporating Key's own geometric relationships makes available conventional text search techniques for two-dimensional geometric searches.

  Thus, the index table mechanism converts the features detected in the image patch into text terms, which represent both the features themselves and their geometric positional relationships. This allows the use of normal text indexes and search methods. For example, the vertically adjacent terms “cat” and “the” are represented by the symbol “cat + the” and may be referred to as “query terms” as will be apparent from this disclosure. Utilizing the normal text search data structure and methodology encourages the MMR method to be tied to the top of Internet text search systems (eg, Google, Yahoo, Microsoft, etc.).

  In the reverse term index table 3422 of this embodiment, each record (record) specifies a candidate area of a page in a document using six parameters. The six parameters are a document identifier (DocID), a page number (PG), an x / y offset (X and Y offsets), and a rectangular area width and height (W and H). The DocID is a unique character string (string) generated based on the time stamp (or other metadata) at the time when the document is printed. However, any character string combining the device ID and the person ID may be used. In any case, the document is identified by a unique DocID and has a record stored in the document index table. The page number is the number of pages corresponding to paper output and starts from 1. The rectangular area is specified by the xy coordinate parameters in the upper left corner in addition to the width and height of the enclosing box in the standardized coordinate system. Various internal document location / coordinate systems will become apparent from this disclosure, and the present invention is not intended to be limited to any particular one.

An example record structure constructed according to one embodiment of the present invention uses a 24-bit DocID and an 8-bit page number, and allows up to 16 × 10 ⁶ documents and 4 × 10 ¹⁰ pages. One unsigned byte for each X and Y offset of the bounding box gives a spatial resolution of 30dpi horizontally and 23dpi vertically (assuming 8.5 "x 11" pages, but other page sizes And / or spatial resolution may be used). Similar handling of the bounding box width and height (for example, one uncoded byte for each of W and H) allows the representation of a region as small as an “i” dot or interval. Or an area as large as the whole page can be expressed (for example, 8.5 ”× 11”). Therefore, 8 bytes per record (3 bytes per DocID, 1 byte per PG, 1 byte per x, 1 byte per y, 1 byte per w, 1 byte per H, 8 bytes total) Can be accommodated.

  The document index table 3424 includes information related to each document. In one particular embodiment, this information includes document related fields in the XML file, which include print resolution, print date, paper size, shadow file name, page image location, and the like. Since the print coordinates are converted into a standardized coordinate system when the document is designated, the search candidate calculation does not include this table. Accordingly, the document index table 3424 receives an inquiry only about candidate areas to be collated. However, the determination includes some information loss in the index—usually because the normalized coordinates are a lower resolution than the print resolution. An alternative uses the document index table 3424 (or higher than the resolution in the standardized coordinate system), if desired, when calculating search candidates.

  In this way, the index table module 3404 effectively provides an image index that allows a content-based search of xy coordinates and objects (eg, document pages) in the object where a given image query appears. To work. Such a combination of image index and relational database 3408 allows the identification of image patches and objects that match the characteristics of the patches (e.g., extraction of "actions" associated with patches, or other content associated with patches). Can be scanned to cause barcodes etc.). Relational database 3408 provides a means for “reverse linking” of patches to features in the index table for other patches of the document. The reverse link provides a way to find features, the recognition algorithm predicts that it will appear to move from one part of the document image to another, and the front-end image analysis algorithm in the MMR system as described above. Improve performance considerably.

Not only the feedback-oriented feature search document and page identity, but also the xy coordinates of the image patch (eg, the xy coordinates relative to the center of the image patch) are input to the feedback-oriented feature search module 3418. The feedback-oriented feature search module 3418 searches the term index table 3422 for records 3423 that are within a given distance from the center of the image patch. For example, the search is facilitated by storing records 3423 for each DocID-PG combination in successive memory blocks in the order of X or Y values. For all records with a given X and Y value, the search is performed by a binary search for the given value and a continuous search from that location (X or Y is how the data is stored) Depending on when it was done and when it was stored.) Typically, this includes the xy coordinates of the M inch ring around the periphery of the patch, which ring has a width of W inches and a height of H inches for a given document and page. Records that are within this ring are identified and their keys or features 3421 are identified by tracing back the pointer. A list of features in the ring and their xy coordinates is reported as shown at 3417 in FIG. 34A. The values of W, H, and M shown in 3415 can be set dynamically by the recognition system based on the size of the input image so that the feature 3417 is outside the input image patch.

  Such properties of the image database system 3400 are advantageous, for example, in eliminating ambiguity among multiple candidates. If the database system 3400 reports more than one document to match the input image patch, the user is instructed to slightly shift the image capture device in a direction that makes the decision clearer. The features in the ring allow the recognition system (eg, fingerprint verification module 226 or other suitable recognition system) to determine which document best matches the document held by the user. For example (where OCR-based features are used, but the content is spread over some geometrically indexed features), the image patch for document A is the word pair “blue-xylophone” May be directly underneath. The image patch for document B may be directly under “blue-thunderbird”. The database system 3400 reports the expected location of these features, and the recognition system places the camera on the top (eg, via the user interface) by the amount specified by the difference between the top of the patch and the y coordinate of the feature. The user can be instructed to move. The recognition system calculates features in the different areas and uses the features from documents A and B to determine which matches best. For example, along with a “dictionary” of features composed of (xylophone, thunder), the recognition system can perform post-processing of OCR results from different regions. The word that best matches the OCR result corresponds to the document that best matches the input image. A specific example of a post-processing algorithm is the generally known spelling correction method (such as that used in word processors and email applications).

  As this example shows, by collating feature descriptions in a way that eliminates the need to perform another database access, the design of the database system 3400 allows multiple candidate ambiguities in a way that the recognition system is efficient. Can be eliminated. An alternative is to process each image independently.

As well as the dynamic initial image generation document and page identity, the xy coordinates of the location of the image patch (eg, the xy coordinate of the center of the image patch) are also entered into the relational database 3408, which is Can be used to retrieve a stored electronic source of a page. The document can be converted as a bitmap image by the document conversion application module 3414. Also, the additional “box size” value provided by module 3414 is used by sub-image extraction module 3416 to extract the portion of the bitmap near its center. This bitmap is an “initial” representation of the predicted display of the image patch and includes an accurate representation of all the features present in the input image. The initial patch is returned as patch feature 3412. This solution overcomes the excessive storage problem required by conventional methods of storing bitmaps by storing compact non-image representations that are later converted to bitmap data on demand.

  Such a storage method is advantageous. This is because it makes possible the use of hypothesis and inspection recognition methods, in which the feature representation derived from the image is used to derive candidate groups, which are then obscured by detailed feature analysis. It will be resolved. Often, it is desirable to determine features from the original images of these candidates, since it is not possible to predict the features that best define alternative candidate groups (disambiguating). For example, an image of the word pair “the cat” is in two database documents, one originally printed in a Times Roman font and the other in a Helvetica font. A check similar to the check whether the input image contains one of these fonts identifies the database document that matches correctly. The correct candidate is identified by comparing the original patch of those documents with the input image patch along with a template matching comparison metric such as Euclidean distance.

  Examples include a relational database 3408 that stores a Microsoft Word “.doc” file (similar methods work with other document formats, such as PCL, pdf., Or Microsoft XML paper specifications XPS, or other such formats, and other such formats are converted to bitmaps by Microsoft Internet Explorer (with installed WinFX components) in the case of conversion applications such as ghost scripts or XPS Is possible.) Document, page, xy coordinates, box dimensions, and system parameters are specified, indicating that the preferred resolution is 600 dots per inch (dpi), and the word application is launched to generate a bitmap image Is possible. This results in a 6600 row and 5100 column bitmap. Additional parameters x = 3 ″, y = 3 ″, height = 1 ″ and width = 1 ″ are databases that have a 600 pixel height and width patch (1800 pixels in the x and y directions from the upper left corner of the page) To return).

Multiple Databases When multiple database systems 3400 are used, each database system includes various document collections, whether two databases are returning the same document or which database returned a candidate that is closer to the input. An initial patch is available to check. If two databases return the same document, possibly with different identifiers 3410 (ie they do not appear to be the same as the original document because they are separately in different databases) and feature 3412, the initial patch is It is almost exactly the same. For example, it can be determined by comparing with the initial patch together with the Hamming distance, and the Hamming distance counts the number of different pixels. If the original document is exactly the same for each pixel, the Hamming distance is zero. If the patches are slightly different, as caused by minor font differences, the Hamming distance will be slightly greater than zero. This can cause a “halo” effect around the edge of the character when image differences are calculated by the hamming operator. Such font differences may be caused by differences in the original conversion application version, operating system versions at the server running the database, printer differences, font collection differences, and the like.

  The initial patch comparison algorithm can be performed on the patch (s) with more than one xy coordinate in two documents. Although they should all be the same, such a sampling process allows redundancy to overcome the representation of differences between database systems. For example, when represented on two systems, one font may look radically different, while another font may look almost the same.

  If more than one database returns a different document than the one that best fits the input image, a pixel-based comparison metric, such as a Hamming distance, compares the initial patch with the input image to determine which is correct.

  An alternative method of comparing results from more than one database is to compare the contents of the accumulator array, which is a geometric distribution (array) of the in-process features reported from each database. It measures. In order to avoid the need to look for the original feature group separately, it is desirable that this accumulator be provided right next to the database. This accumulator should also be independent of the contents of the database system 3400. In the example shown in FIG. 34A, an activity array 3420 is output. Two activity arrays can be compared by measuring their distribution in the array.

  More specifically, two or more databases may be the same document, possibly with different identifiers 3410 (ie, they do not appear to be the same as the original document because they are separately in different databases)) and feature 3412. The activity array 3420 from each database is almost exactly the same. For example, it can be determined by comparing with the initial patch together with the Hamming distance, and the Hamming distance counts a different number of pixels. If the original document is exactly the same for each pixel, the Hamming distance is zero.

  If more than one database returns a document that is different from the one that best fits the input features, their activity arrays 3420 can be compared to determine which document “best” fits the input image. An activity array that appropriately matches an image patch includes high value clusters that are clustered near the location where the patch appears. An activity array that only inappropriately matches an image patch will contain randomly distributed values. There are many well-known methods for measuring the dispersibility and randomness of an image such as entropy. Such an algorithm is applied to the activity array 3420 to obtain an indicator that indicates the presence of a cluster. For example, the entropy of an activity array 3420 that includes clusters corresponding to image patches is significantly different from the entropy of an activity array 3420 that has randomly distributed values.

  Furthermore, it should be noted that each client 106 may access multiple database systems 3400 at any time, and the contents of the databases do not necessarily compete with each other. For example, a company may own both publicly accessible patches and private ones (each associated with one document) for that company. In such a case, the client 106 maintains a list of databases D1, D2, D3,... (Databases are queried in that order) and an activity array 3420 and identifier 3410 coupled to a consolidated display for the user. Is generated. A given client device 106 may display available patches from all databases, or may allow a user to select a portion of a database (eg, only D1, D3, and D7). Or just display the patches in those databases. By subscribing to a service, the database may be added to the list, or the database may be made available wirelessly if the client device 106 is at a location. This is because the database is one of several that are loaded on the client device 106, because a user is currently authenticated using that device, or the device is operating in a mode. Even because they are. For example, several databases may result from a particular client device turning its audio speaker on or off, or because a peripheral device such as a video projector is currently attached to the client. May be available.

Referring further to the action diagram 34A, the MMR database 3400 receives certain actions with a group of features from the MMR feature extraction module 3402. Actions specify commands and parameters. In such an example, the command and its parameters determine the returned patch characteristics 3412. For example, actions may be received in a format that includes http that can be easily converted to text.

  The action processor 3413 receives the xy coordinates in the page, the page and the identity of the document determined by the evidence storage module 3406. The action processor also receives commands and their parameters. The action processor 3413 is programmed or constructed to convert commands into instructions (instructions) to extract or store data using a relational database 3408 at a location corresponding to a given document, page and xy coordinates. To do.

  In one such embodiment, the commands include: RETRIEVE, INSERT_TO <DATA>, RETRIEVE_TEXT <RADIUS>, TRANSFER <AMOUNT>, PURCHASE, PRISTINE_PATCH <RADIUS [DOCID PAGEID XY DPI]> and ACESS_DATABASE <DBID> . Each of these will be described below.

  RETRIEVE retrieves data linked to xy coordinates in a given document page. The action processor converts the RETRIEVE command into a relational database query and extracts data stored in the vicinity of the xy coordinates. This may require issuing more than one database query to search for the area around that xy coordinate. The retrieved data is output as a patch feature 3412. A specific application of the RETRIEVE command is a multimedia browsing application, which extracts a video clip and a dynamic information object (for example, an electronic address from which current information can be extracted). The extracted data may include a menu that specifies subsequent steps to be executed by the MMR device. The extracted data may be static data, which can be displayed on the phone (or on other display devices) like JPEG images or video clips. A parameter for determining an area to search for patch features may be provided in the RETRIEVE command.

  INSERT_TO <DATA> inserts <DATA> into the xy coordinates specified by the image patch. The action processor 3413 converts the INSERT_TO command into an instruction for a relational database, and adds data to the designated xy coordinates. A confirmation notice that the INSERT_TO command has been successfully completed is returned as a patch feature 3412. A specific application of the INSERT_TO command is a software application on the MMR device that allows the user to attach data to any xy coordinate in a passage of text. The data can be static multimedia such as JPEG images, video clips or audio files, etc., but can also be arbitrary electronic data such as menus that specify actions associated with a given location.

  RETRIEVE_TEXT <RADIUS> extracts text within the radius (<RADIUS>) of xy coordinates determined by the image patch. <RADIUS> may be specified, for example, as the number of pixels in the image space, or may be specified as the number of characters in the words around the xy coordinates determined by the evidence storage module 3406. <RADIUS> may relate to the text object being analyzed. For this particular example, the action processor 3413 converts the RETRIEVE_TEXT command into a relational database query and extracts the appropriate text. If <RADIUS> specifies a text object to be analyzed, the action processor only returns the text object to be analyzed. If the text object to be analyzed is not found near the specified xy coordinates, the action processor returns a null index. In the alternative, the action processor invokes a feedback-oriented feature search module to retrieve text that is within the radius of a given xy coordinate. A text string is returned as the patch feature 3412. The selective data associated with each word in the text string includes an xy bounding box in the original document. A specific application of the RETRIE_TEXT command is to select a text phrase from a printed document and include it in a separate document. This can be used, for example, to compose a presentation file in an MMR system (eg, in PowerPoint format).

  TRANSFER <AMOUNT> retrieves the entire document or a portion of the data linked to the document in a format that can be loaded into another database. <AMOUNT> specifies the number and type of data to be retrieved. If <AMOUNT> is ALL, the action processor 3413 issues a command to the database 3408 to retrieve all data associated with the document. Examples of such commands include DUMP or Unix® TAR. If <AMOUNT> is SOURCE, the original source file for the document is retrieved. For example, the action processor retrieves a word file of a printed document. If <AMOUNT> is BITMAP, a JPEG compressed version (or other commonly used format) of the bitmap of the print document is retrieved. If <AMOUNT> is PDF, the PDF representation of the document is retrieved. The retrieved data is output as a patch feature 3412 in a format known to the calling application by the command name. A specific application of the TRANSFER command is “document grabber”, which allows a user to transfer a PDF representation of a document to an MMR device by image processing a small area of text.

  PURCHASE retrieves the product specification linked to the xy coordinates in the document. The action processor 3413 first executes a series of one or more RETRIE commands to obtain a product specification near a given xy coordinate. The product specification includes, for example, a vendor name, a product identity (for example, a stock number), a vendor electronic address, and the like. Product specifications are obtained in preference to other data types that may be located nearby. For example, when jpeg is stored in the xy coordinates determined by the image patch, the product specification of the nearest nearest neighbor is taken out instead. The retrieved product specification is output as a patch feature 3412. The specific application of the PURCHASE command relates to advertising on printed documents. The software application on the MMR device receives the product specification associated with the advertisement, adds the user's personal identification information (eg name, shipping address, credit card number, etc.) and then specifies it Send to a vendor with an electronic address.

  PRISTINE_PATCH <RADIUS [DOCID PAGEID XY DPI]> obtains an electronic representation of a specified document and obtains an image patch within a radius RADIUS centered on xy. RADIUS originally specifies the radius of a circle, but may specify a rectangular patch (for example, a height of 2 inches and a width of 3 inches). Alternatively, the entire document page may be specified. The information of (DocID, PG, x, y) may be given explicitly as part of the action, or may be derived from the image of the text patch. Action processor 3413 retrieves the original representation of the document from relational database 3408. The representation may be a bitmap or a displayable electronic document. The original representation is communicated to the document rendering application 3414, converted to a bitmap (with resolution that can be given in parameters DPI as dots per inch), and given to the sub-image extractor 3416 to retrieve the desired patch . The patch image is returned as patch feature 3412.

  ACESS_DATABASE <DBID> adds the database 3400 to the client 106 database list. The client can query not only the databases that are in the current list, but also the databases that have just been added. DBID specifies the file or remote network for the specified database.

Index Table Generation Method FIG. 35 illustrates a method 3500 for generating an MMR index table according to one embodiment of the present invention. This method can be executed, for example, from the database system 34002 of FIG. 34A. In such an embodiment, the MMR index table module 3404 (or a detailed alias module) generates an MMR index table from the scanned or printed document. The generation module is in software, hardware (e.g., gate level logic), firmware (e.g., a microcontroller built with an embedded routine that performs the method), or some combination thereof. It can be realized in the same manner as other modules described in the above.

  The method includes receiving 3510 a paper document. The paper document can be any document, for example a memo (e.g. business-related memos, personal letters), product labels (e.g. canned goods, medicines, boxed electronic devices), product specifications (e.g. Snow browsers, computer systems, manufacturing systems), product brochures or advertising materials (eg cars, boats, vacation resorts), service description materials (eg internet service providers, cleaning services), books, magazines and other publications A page or more, a page printed from a website, a handwritten memo, a memo captured and printed from a whiteboard, a page printed from some processing system (eg, desktop, portable computer, camera, smartphone, remote terminal).

  The method continues with generating 3512 an electronic representation of the paper document, the electronic representation including xy coordinates of features shown in the document. The target (search target) feature may be, for example, in a document such as an individual word, a character (s), and / or a character. For example, when an original document is scanned, it is first OCR processed to extract words (or other searchable features) and their xy coordinates (eg, by the document fingerprint matching module 226 of the scanner 127). Extracted). When the original document is printed, the indexing process (indexing process) can be performed in the XML format of all character fonts, sizes, xy bounding boxes (or other searchable features) (eg, printer 116 printer). Receive accurate representation (by driver 316 processing). In this case, index table generation begins at step 3514 because the electronic document is received with the accurately specified xy feature coordinates (eg, by printer driver 316). Formats other than XML will also be apparent from the disclosure of this application. Electronic documents such as Microsoft Word, Adobe Acrobat and PostScript can be entered into the database by "printing" them to the printer driver, and the output of the printer driver must be directed to a file and a paper document must be created Not to be. This causes the generation of the XML file structure shown below. In all cases, in addition to the original document format (word, acrobat, postscript, etc.), XML is assigned to an identifier (doc i for the i-th document added to the database), and a method that can be searched later by that identifier In addition to being stored in the relational database 3408, it is based on other “metadata” characteristics of the document, including time captured, date printed, application that caused printing, output file name, and so on.

A specific example of an XML file structure is as follows:

  In certain embodiments, the word may include any character such as az, AZ, 0-9, and @% $ #, all others are delimiters. The original description of the .xml file can be generated by the print capture software used in the indexing process (eg, it is executed on a server such as database 320 server). As new documents are captured by the system, the actual format will continually evolve and include more elements.

  The original sequence of text received by the print driver (eg, print driver 316) is preserved and the logical word structure is constrained based on the punctuation symbols except for “_ @ & $ #”. Using an XML file as input, the index table module 3404 first attempts to group sequences into logical lines by looking at page boundaries and examining the amount of overlap between two consecutive sequences. In one particular embodiment, if the two sequences overlap less than half their average height, a heuristic is used that a line break is occurring. Such a heuristic works well with typical text documents (eg, Microsoft Word). For htm pages with complex layouts, additional geometric analysis may be required. However, it is not essential to retrieve a complete semantic document structure as long as a consistent indexing term can be generated by the query process.

  Based on the electronic document structure of the paper document, the method continues to index (3514) all of the searchable features for all pages of the paper document. In one particular embodiment, this step includes indexing the positions of all horizontally and vertically adjacent pairs on all pages of the paper document. As described above, horizontally adjacent words are pairs of adjacent words in a line (row). Vertically adjacent words are words that are adjacent in vertically aligned lines. Other multidimensional forms in the page may be used as well.

  The method includes storing (3516) a patch feature associated with each search target feature. In one particular embodiment, the patch features include actions associated with the patch and are stored in a relational database. As described above, combining such image index and storage means allows for the discovery of objects that match the image patch and the characteristics of the patch. The feature can be any data related to the path, such as metadata. Features can be scanned to cause, for example, actions that perform a specific function, links that can be selected to give access to other content related to the patch, and / or extraction of other content related to the patch Alternatively, it may include a barcode or the like that can be processed.

  A more accurate definition is given for search term generation and only a portion of the line structure is observed. In the case of horizontally adjacent pairs, a query term is formed by concatenating the word and a “-” separator. Vertical pairs are connected using “+”. Words can be used in their original form and preserve capitalization (case) if desired (this creates a more specific term, but makes such a subtle situation It will create a larger index with additional query items to consider.) This indexing method allows the same search method to be applied to horizontal or vertical word pairs, or a combination of both. The ability to identify a term is described by the inverse document frequency for any case.

Evidence Accumulation Method FIG. 36 illustrates a method 3600 according to one embodiment of the present invention for calculating a ranked group of documents, pages and location hypotheses of documents to be searched. The method can be performed, for example, on the database system 3400 of FIG. 34A. In such an embodiment, the evidence accumulation module 3406 calculates hypotheses (candidates) using data from the index table module 3404 as described above.

  The method begins by receiving 3610 a document image to be searched (such as a large document image or an image patch of the entire document image). Following the generation of one or more query terms (3612), the method captures a two-dimensional positional relationship between objects in the searched document image. In one particular embodiment, the query terms are generated by a feature extraction process that generates horizontal and vertical word pairs as described with reference to FIG. 34B. However, as will become apparent from the present disclosure, any number of the feature extraction processes described herein can be used to generate a query term, and the two-dimensional positional relationship between objects in the document image to be searched is determined. To capture. For example, the same feature extraction method used to build the index of method 3500 can be used to generate a query term as described with reference to step 3512 (an electronic representation of a paper document). To generate.) Further, a two-dimensional form of query terms may be applied to each query term (eg, a first query term that is a horizontal word pair and a second query term that is a vertical word pair), or a group of search terms. Note that it may be applied (eg, one query term representing both horizontal and vertical objects in the document to be searched).

  The method continues by looking for each query term in the term index table 3422 and extracting a list of locations associated with each query term (3614). For each location, the method continues by generating a number of regions containing that location (3616). After all queries have been processed, the method includes identifying the region that best matches all query terms (3618). In such an embodiment, the scores for all candidate regions are increased by some weight (eg, based on how well each region matches all query terms). The method continues by determining (3620) whether the identified region meets a predetermined match criterion (eg, based on a predetermined match threshold). If so, the method follows (3622) following verifying the region to match the document image being searched (e.g., the page containing the most region is accessed or used. ). Otherwise, the method continues to reject the area (3624).

  The word pairs are stored in the term index table 3422 along with the coordinates in the “normalized” coordinate space. This provides a unity between the resolutions of the various printers and scanners. In one particular embodiment, an 85 × 110 coordinate space is used for 8.5 ″ × 11 ″ pages. In that case, all word pairs are identified at a place in the 85 × 110 space.

  To improve the search efficiency, a two-stage process may be performed. The first step involves finding the page that is most likely to contain the input image patch. The second step involves calculating the xy coordinates likely to be the center of the patch in the page. Such a method leads to the risk that the truly best match will be missed in the first step. However, such a possibility with a small indexing space is rare. Thus, depending on the size of the index and the desired performance, such an efficient improvement method can be used.

In one such embodiment, the following algorithm is used to find a page, which is most likely to contain a word pair that is detected in the input image patch.

  This method adds the document frequency inverse function (idf) of each word pair to the accumulator indexed by the document and page in which the word pair occurs, and num_docs (wp) returns the number of documents containing the word pair wp. The accumulator is implemented by the evidence accumulation module 3406. If the maximum value in the accumulator exceeds the threshold, the page that best matches the patch is output. Thus, the algorithm operates to identify the page that best matches the word pair in the query. Alternatively, the Accum array is sorted (sorted) and the top N pages are reported as “best N” pages that match the input image.

The following evidence accumulation algorithm accumulates evidence regarding the position of an input image patch within a page, according to one embodiment of the present invention.

  The algorithm operates to identify a cell in the 85 × 110 space, which is most likely the center of the input image patch. In the example shown here, the algorithm does this by adding weights to cells in a fixed area (also called a zone) around each word pair.

  An extent function is given to the x, y pair, and the range function returns the minimum and maximum values of the enclosing fixed size region (typically 1.5 "high and 2" wide). ). The range function takes into account the boundary conditions and ensures that the value returned does not go out of the accumulator (i.e., not in a region that is less than 0, x is greater than 85, or y is greater than 110). ). The max distance function (maxdist function) finds the maximum Euclidean distance between two points in a bounding box described by bounding box coordinates (minx, maxx, miny, maxy). A weight determined by the product of the inverse document frequency function of the word pair and the normalized geometric distance between the cell and the center of the zone is calculated for each cell in the zone. This weights cells closer to the center and higher than cells far away. After all word pairs have been processed with the algorithm, the Accum2 array looks for the cell with the maximum value. If it exceeds the threshold, the coordinates are reported as the location of the image patch. The activity array stores the accumulated norm_dist value. Since they are not scaled with idf, they do not consider the number of documents in the database that contain a particular word pair. However, they provide a two-dimensional image representation for the xy coordinates that best fit a given group of word pairs. In addition, the activity array entries are independent of the documents stored in the database. This data structure normally used internally may be output (3420).

ワードペアの場所とゾーンの中心との間のユークリッド距離が計算され、そのユークリッド距離と、計算される可能性のある最大距離との間の差分が返される。 The normalized geometric processing is calculated as shown below according to one embodiment of the present invention.

The Euclidean distance between the word pair location and the center of the zone is calculated and the difference between the Euclidean distance and the maximum distance that can be calculated is returned.

  After the word pair is processed with the evidence accumulation algorithm, the Accum2 array is searched for the cell with the maximum value. If the value exceeds a predetermined threshold, the coordinates are reported as the center location of the image patch.

MMR Printing Architecture FIG. 37A shows a functional block diagram of an MMR component according to one embodiment of the present invention. The main MMR components include a computer 3705 with an associated printer 116 and / or shared document annotation (SDA) server 3755.

  Computer 3705 is any standard desktop, laptop or network computer known in the art. In one embodiment, the computer is the MMR computer 112 described with reference to FIG. 1B. User printer 116 is any standard home, office or commercially available printer known in the art. The user printer 116 generates a print document 118, which is a paper document formed of one or more pages.

  The SDA server 3755 is a standard networked or centralized computer that holds various files, applications and / or information regarding shared annotations. For example, shared annotations associated with web pages or other documents are stored on the SDA server 3755. In this example, the annotation is data or instructions used in the MMR described above. The SDA server 3755 is accessible via a network connection in one embodiment. In one embodiment, SDA server 3755 is networked media server 114 described with reference to FIG. 1B.

  Computer 3705 has various components, all or some of which are optional depending on the various embodiments. In one embodiment, computer 3705 includes source file 3710, browser 3715, plug-in 3720, symbolic hotspot description 3725, modified file 3730, capture module 3735, page_desc.xml3740, hotspot.xml3745, data store 3750, SDA server Includes 3755 and MMR printing software 3760.

  Source file 3710 represents some source file that is an electronic representation of a document. Specific examples of source files 3710 include hypertext markup language (HTML) files, Microsoft® Word® files, Microsoft® PowerPoint® files, simple text files, portable document formats ( PDF) file etc. are included. As described above, documents received by browser 3715 often originate from source file 3710. In one embodiment, source file 3710 is equivalent to source file 310 described with reference to FIG.

  Browser 3715 is an application that provides access to data associated with source file 3710. For example, browser 3715 may be used to retrieve web pages and / or documents from source file 3710. In one embodiment, the browser 3715 is the SD browser 312 314 described with reference to FIG. In one embodiment, browser 3715 is an Internet browser such as Internet Explorer.

  Plug-in 3720 is a software application that provides an authoring function. The plug-in 3720 may be a stand-alone software application, or may be a plug-in that runs on the browser 3715. In one embodiment, plug-in 3720 is a computer program that interacts with an application such as browser 3715 to provide the specific functionality described above. Plug-in 3720 performs conversions and other modifications to web pages and documents displayed on browser 3715 according to various embodiments. For example, plug-in 3720 creates a hot spot by enclosing a hot spot indicator with individually identifiable fiducial marks, returns a “markup” version of the HTML file to browser 3715, and the document displayed in browser 3715 A conversion rule is applied to a part of the document, and a shared annotation for the document displayed in the browser 3715 is extracted and / or received. Further, the plug-in 3720 may perform other functions as described above, such as creating a modified document, creating a symbolic hotspot description 3725, and the like. Plug-in 3720 supports the method described in FIGS. 38, 44, 45, 48 and 50A-B in connection with capture module 3735.

  The symbolic hot spot description 3725 is a file that identifies a hot spot in a document. The symbolic hot spot description 3725 specifies the number of hot spots and the content. In this example, the symbolic hot spot description 3725 is stored in the data store 3750. A specific example of a symbolic hot spot description is described in more detail in FIG.

  The modified file 3730 is a document and web page created as a result of modification and conversion of the source file 3710 by the plug-in 3720. For example, the markup HTML file described above is an example of a modified file 3730. As will become apparent from the present disclosure, the modified file 3730 is optionally returned to the browser 3715 for display to the user.

  The capture module 3735 is a software application that performs feature extraction and / or coordinate capture on a printed representation of a document so that the layout of characters and graphics on a printed page can be retrieved. Layout—That is, a two-dimensional array of text on a printed page may be captured automatically when printed. For example, the capture module 3735 executes all character and drawing print commands, and also acquires and records all characters and / or xy coordinates of the image and other characters in the printed representation. According to one embodiment, the capture module 3735 may be a print capture DLL as described herein, a forwarding dynamically linked library (DLL) that allows additions, or a modified version of an existing DLL, etc. It is. A more detailed description of the function of the acquisition module 3735 is given in connection with FIG.

  One skilled in the art will recognize that the capture module 3735 is coupled to the output of the browser 3715 to capture data. Alternatively, the functions of the capture module 3735 may be performed directly in the printer driver. In one embodiment, capture module 3735 is equivalent to a PD capture module as described with respect to FIG.

  page_desc.xml 3740 is an Extensible Markup Language (XML) file in which text related output is written for function calls processed by the text related capture module 3735. In addition to hotspot information, printer port name, browser name, date and time of printing, dots per inch (dpi) and resolution (res) information, page_dec.xml3740 contains all text printed for each word and for each character. About document coordinate information. For example, page_dec.xml3740 is stored in the data store 3750. The data store 3750 is equivalent to the MMR database 3400 described with reference to FIG. 34A. 42A-B show in detail a specific example of page_dec.xml3740 in the case of an HTML file.

  hotspot.xml 3745 is an XML file created when a document is printed (for example, as described above, by the operation of the print driver 316). hotspot.xml is the result of merging the symbolic hotspot description 3725 and page_dec.xml3740. hotspot.xml includes hot spot identification information such as the number of hot spots, coordinate information, size information, and hot spot content. FIG. 43 shows a specific example of the hotspot.xml file.

  Data store 3750 is any database known in the art for storing files, but modified to utilize the methods described herein. For example, according to one embodiment, data store 3750 includes source file 3710, symbolic hotspot description 3725, page_dec.xml3740, rendered page layout, shared annotations, imaged document, hotspot definition and Stores feature expressions and the like. In one embodiment, the data store 3750 is equivalent to the document event database 320 described with reference to FIG. 3 and is equivalent to the database system 3400 described with reference to FIG. 34A.

  The MMR printing software 3760 is software that supports the MMR printing process described here, and is executed by, for example, the components of the computer 3705 described above. The MMR printing software 3760 is described in further detail below with reference to FIG. 37B.

  FIG. 37B shows a group of software components included in the MMR printing software according to one embodiment of the present invention. It should be understood that all or part of the MMR printing software 3760 may be included in the computer 112,905, capture device 106, network media server 114, and other servers as described above. Although the MMR printing software 3760 is currently described as including these various components, those skilled in the art will appreciate that the MMR printing software 3760 may include any number from one to all of these components. Will recognize. The MMR printing software 3760 includes a conversion module 3765, an embedded module 3768, an analysis module 3770, a conversion module 3775, a feature extraction module 3778, an annotation module 3780, a hot spot module 3785, a representation / display module 3790, and a storage module 3795.

  The conversion module 3765 allows the source document to be converted into an image document, and feature representations can be extracted from the image document, and the conversion module is one means of doing it.

  The built-in module 3768 is one means of allowing and embedding a mark corresponding to the hot spot indicator of the electronic document. In one particular embodiment, the embedded marks indicate the hot spot start point and hot spot end point. Alternatively, a predetermined area around the embedded mark may be used to identify a hot spot in the electronic document. Various such marking methods can be used.

  The analysis module 3770 is one means of enabling and performing analysis of the electronic document (sent to the printer) for the mark indicating the starting point of the hot spot.

  The conversion module 3775 is one way to make certain conversion rules applicable to a part of an electronic document and to do so. In one particular embodiment, the portion is a stream of characters between a mark indicating the start point of the hot spot and a mark indicating the end point of the hot spot.

  The feature extraction module 3778 is one means of enabling and extracting the coordinates and features corresponding to the printed representation of the document and hotspot. Coordinate capture includes retrieving print commands using a forwarding dynamic link library and analyzing the printed representation for a portion of the coordinates corresponding to the hotspot or transformed character. The feature extraction module 3778 enables the functionality of the capture module 3735 according to one embodiment.

  Annotation module 3780 is one means of enabling and receiving shared annotations and accompanying indicator portions of documents associated with the shared annotations. Receiving shared annotations includes receiving annotations from the end user and from the SDA server.

  Hotspot module 3785 is one means of allowing and doing one or more clips to be associated with one or more hotspots. The hot spot module 3785 allows the creation of a hot spot definition by first specifying the location of the hot spot in the document and determining the clip associated with the hot spot.

  The representation / display module 3790 is a means for rendering or displaying a document or a printed representation of a document.

  Storage module 3795 is one means of enabling and storing various files including page layouts, image processed documents, hotspot definitions and feature representations.

  It is not essential that the software portion 3765-3795 is an individual software module. The software configuration shown is merely exemplary and other configurations are envisioned within the scope of the present invention, as will be apparent from the disclosure.

Embedding Hot Spots in a Document FIG. 38 shows a flowchart of a method for embedding hot spots in a document according to one embodiment of the present invention.

  According to the method, a mark associated with an indication of a hot spot in the document is embedded in the document (3810). In one embodiment, a document containing a hotspot indication location is received for display in a browser, for example, a document is received at the browser 3715 from a source file 3710. Hotspots include, in addition to electronic data, some text or other document objects such as graphics or photos. The electronic data may include multimedia such as audio or video, or may represent a group of steps performed at the capture device when the hotspot is accessed. For example, if the document is a hypertext markup language (HTML) file, the browser may be Internet Explorer and the instruction may be a uniform resource locator (URL) in the HTML file. FIG. 39A shows an example of such an HTML file 3910 with a URL 3920. FIG. 40A shows the text of the HTML file 3910 of FIG. 39A when displayed on a browser 4010 such as Internet Explorer.

  In order to embed the mark in (3810), the plug-in 3720 of the browser 3715 surrounds each hot spot indication position together with each identifiable reference mark, and creates a hot spot. In one embodiment, plug-in 3720 modifies the document displayed in browser 3715 (eg, HTML displayed in Internet Explorer following the above) and bundles hotspot indication locations (eg, URLs). Insert a mark or tag. The mark may be undetectable by an end user viewing the document in the browser 3715 or in a printed version of the document, but is detectable with a print command. In this example, a new font referred to as MMR Courier New is used to add the start and end of fiducial marks. In the MMR Courier New font, general glyphs or dot patterns for characters “b”, “e” and numbers are represented by blank spaces.

  Further referring to the HTML page example shown in FIGS. 39A and 40A, the plug-in 3720 embeds the reference mark “b0” (“here”) at the start position of the URL and the reference mark “e0” at the end position of the URL. The hot spot is indicated together with the identifier “0” (3810). Since b, e and numeric characters are displayed as spaces, the user sees little or no change in the display of the document. Further, the plug-in 3720 creates a symbolic hot spot instruction (information) 3725 indicating these marks as shown in FIG. The symbolic hot spot indication 3725 identifies the hot spot number as 0, which corresponds to 0 in the “b0” and “e0” reference marks. In this example, the symbolic hot spot instruction 3725 is stored in the data store 3750, for example.

  As shown in FIG. 39B, the plug-in 3720 returns an HTML markup version 3950 to the browser 3715. Markup HTML 3950 surrounds the fiducial mark with a span tag 3960, which turns the font into a 1-point MMR Courier New. Since b, e, and numeric characters are shown as spaces, the user sees little or no change on the document display. Markup HTML 3950 is an example of a modified file 3730. These examples use a single page model for simplicity, but multiple page models may use the same parameters. For example, when a hot spot crosses a page boundary, the hot spot has a reference mark corresponding to each page position, and each hot spot identifier is the same.

  Next, in accordance with the print command, coordinates corresponding to the print expression and the hot spot are acquired (3820). In one embodiment, the capture module 3735 “taps” text and drawing commands in the print command. The capture module 3735 executes all text and drawing commands, and further acquires and records the xy coordinates and other characters of all characters and / or images in the printed representation. In this example, capture module 3735 refers to a device context (DC) for a printed representation, where DC is an operation handle to the structure of the printed representation, which structure is output text and / or image depending on the output format. Attributes (eg, printer, window, file format, memory buffer, etc.). In the process (3820) of capturing coordinates for printed representations, hot spots are easily identified using fiducial marks embedded in HTML. For example, when a start mark is encountered, the xy coordinates of all characters are recorded until an end mark is found.

  In one embodiment, the capture module 3735 is a forwarding DLL referred to as a “print capture DLL” that allows additions or modifications to the functionality of an existing DLL. The forwarding DLL shows the client exactly the original DLL, but additional code (tap) is added to all or part of the function before the call is forwarded to the searched (original) DLL. It is done. In this example, the print capture DLL is a forwarding DLL for the Windows Graphics Device Interface (Windows® GDI) DLL gdi32.dll. gdi32.dll has over 600 output ports, all of which need to be transferred. A printable DLL, referred to as gid32_mmr.dll, allows the client to capture printouts from any Windows application that uses the DLL gdi32.dll for drawing and runs on the local computer even when printing to a remote server It only needs to be done.

  According to one embodiment, gdi32_mmr.dll is renamed gdi32.dll and copied to C: \ Windows \ system32, causing it to monitor printing from almost the entire Windows application. In another embodiment, gdi32_mmr.dll is renamed gdi32.dll, copied to the application's home directory, and printing for the application is monitored. For example, C: \ Program Files \ Internet Explorer monitors Windows XP Internet Explorer. In this example, only this application (eg, Internet Explorer) automatically invokes the print capture DLL function.

  FIG. 44 shows a flow chart of a process according to one embodiment of the present invention used in the forwarding DLL. The print capture DLL dgi32_mmr.dll first receives a function call directed to gdi32.dll (4405). In one embodiment, gdi32_mmr.dll receives all function calls directed to gdi32.dll. gdi32.dll monitors about 200 out of approximately 600 function calls, and these are functions that affect the appearance of pages printed in some way. Next, the print capture DLL checks whether the received call is a function call to be monitored. If the received call is not a monitored function call, the call bypasses steps 4415 to 4435 and is forwarded to gdi32.dll (4440).

  If the received call is a monitored function call, the method then determines whether the function call specifies a “new” printer device content (DC) (ie, the printer DC has been received in the past). To check). This is confirmed by inspecting the printer DC against the internal DC table. As described above, DC includes not only drawing settings such as fonts and colors, but also search targets to be drawn (may be a printer, a memory buffer, etc.). All drawing processes (for example, LineTo (), DrawText (), etc.) are executed on the DC. If the printer DC is not new, a memory buffer corresponding to the printer DC already exists and step 4420 is skipped. If the printer DC is new, a memory buffer DC corresponding to the new printer DC is created. This memory buffer DC reflects the appearance of the page to be printed, and in this example is equivalent to the above printed representation. Thus, when a printer DC is added to the internal DC table, a memory buffer DC (and memory buffer) of the same dimension is created and associated with the printer DC in the internal DC table.

  Next, Gdi32_mmr.dll checks whether the call is a text-related function call (4425). About 12 of the 200 gdi32.dll calls monitored are text related. If it is not text-related, step 4430 is skipped. If the function call was text related, the text related output is written to an xml file (4430), which is referred to as page_dec.xml3740 as shown in FIG. 37A.

  42A and 42B illustrate page_dec.xml3740 of the example HTML file 3910 described with reference to FIGS. 39A and 40A. Page_dec.xml 3740 includes coordinate information for all printed text by word 4210 (eg, Get), by x, y, width and height, and by character 4220 (eg, G). All coordinates are in dots, which are like pixels in a printer, and unless otherwise defined, are coordinates for the upper left corner of the page. Page_dec.xml 3740 includes hot spot information such as a start mark 4230 and an end mark 4240 in the form of “sequence”. For hot spots across page boundaries (for example, hot spots from page N to page N + 1), the hot spots appear on both pages (N and N + 1), and in both cases the hot spot identifier is the same. is there. In addition, other information is included in page_dec.xml3740, which includes the printer port name 4250 (created in addition to the number of dots per inch (dpi) and resolution (res) of page 4280 and printable area 4290. Such as the name 4260 of the browser 3715 (or application) and the print date and time 4270.

  Referring back to Figure 44, following confirmation that the call was not text-related, or following writing text-related output to page_dec.xml3740, gdi32_mmr.dll is a function call in the DC memory buffer. Is executed (4435). This step 4435 provides output to the printer and also provides output to the local computer's memory buffer. As the number of pages increases, the contents of the memory buffer are compressed and written in JPEG and PNG formats. The function call is then forwarded to gdi32.dll (4440) and executed as if normally made.

  Referring to FIG. 38, a page layout having a printed representation including hot spots is represented (3830). In one embodiment, the representation 3830 includes printing the document. FIG. 40B shows an example of a print version 4011 of the HTML file 3910 of FIGS. 39A and 40A. Note that the fiducial mark is not visually perceptible to the end user. The expressed layout is stored in, for example, the data store 3750.

  According to one embodiment, the print capture DLL merges the data in symbolic hotspot description 3725 and page_dec.xml3740 into hotspot.xml3745 as shown in FIG. 43, for example as shown in FIGS. 42A-B. . In this example, hotspot.xml3745 is created when the document is printed. The example of FIG. 43 shows that hotspot 0 is at x = 1303, y = 350, 190 pixels wide and 71 pixels high. Hotspot content is shown at http://www.richo.com.

  According to an alternative to the capture module 3820, the filter XPS (XML Print Specification) in the Microsoft printer driver is commonly known as “XPSDrv filter”, receives text drawing commands, and sets the page_dec.xml file above. Create as follows.

Visually Visible Hot Spots FIG. 45 shows a flowchart of a method according to one embodiment of the present invention for converting characters corresponding to hot spots in a document. This method is a method that shows both end-users and MMR recognition software, and modifies a printed document with hot spots.

  First, an electronic document to be printed is received as a character stream (4510). For example, the document may be received 4510 by a printer driver or software module that can screen the character stream. In one embodiment, a document is received at browser 3715 from source file 3710. FIG. 46 shows an example electronic version of a document 4610 according to one embodiment of the present invention. Document 4610 in this example includes two hot spots, one related to “are listed below” and one related to “possible prior art”. Hot spots are not visually perceivable by the end user in one embodiment. The hot spot may be set by the coordinate capturing method described with reference to FIG. 38 or by some other method described above.

  The document is analyzed for a start mark indicating the beginning of the hot spot (4520). The start mark may be a reference mark as described above, or some other individually identifiable mark that identifies a hot spot. Once a start mark is found, certain conversion rules are applied to a portion of the document (4530), i.e., to characters after the start mark until an end mark is found. The conversion rules cause visible modifications to the portion of the document corresponding to the hot spot, in one embodiment, for example, by modifying the font or color of the character. In this example, an original font such as Times New Roman is converted to another known font such as OCR-A. In another example, the text is represented in another font color, such as blue # F86A. The process of converting the font is similar to the process described above in one embodiment. For example, if the document is an HTML file 4610 and encounters a fiducial mark in the document 4510, the font is replaced in the HTML file.

  The conversion step according to one embodiment is performed by a plug-in 3720 to the browser 3715, resulting in a modified document 3730. FIG. 47 shows an example of a printed modified document according to one embodiment of the present invention. As shown, hot spots 4720 and 4730 are visually distinguishable from the rest of the text. In particular, hot spot 4720 is visually distinguishable with different fonts, and hot spot 4730 is visually distinguishable with different colors and underlines.

  The document with the converted portion is then represented in a page layout that includes the electronic document and the location of the hotspot within the electronic document. In one embodiment, rendering the document includes printing the document. In one embodiment, the representation includes performing feature extraction (according to any of the methods as described above) on the document with the transformed portion. In one embodiment, the feature extraction includes capturing page coordinates corresponding to the electronic document in response to the print command. The electronic document is then analyzed for a portion of the coordinates corresponding to the converted character. In one embodiment, the acquisition module 3735 of FIG. 37A performs feature extraction and / or coordinate acquisition.

  The MMR recognition software preprocesses all images using the same conversion rules. First, search for text that complies with the rules (for example, those of OCR-A or Blue # F86A), and then apply the normal recognition algorithm.

  This aspect of the present invention is advantageous in that it uses a very simple image processing routine, thus significantly reducing the computational burden on the MMR recognition software and eliminating most of the computational overhead. In addition, if the bounding box is on a part of the document, for example as described with reference to FIGS. 51A-D, the majority of alternatives that may apply may be removed from the selection to eliminate feature extraction. Accuracy can be improved. Further, the visible modification of the text indicates which text (or which other document object) is part of the hot spot.

Shared Document Annotation FIG. 48 shows a flowchart of a shared document annotation method according to one embodiment of the present invention. The method allows users to annotate documents in a shared environment. In the embodiment described below, the shared environment is a web page viewed by various users, but in the other embodiment, the shared environment is shared resources such as a work group. Any environment is acceptable.

  According to the method, the source document is displayed (4810) in a browser such as browser 3715, for example. In one embodiment, the source document is received from a source file 3710, and in another embodiment, the source document is a web page received over a network, such as an internet connection. To illustrate using that example web page, FIG. 49A shows a source web page sample 4910 in a browser according to one embodiment of the present invention. In this example, web page 4910 is an HTML file of a game related to a general children's book character-Jerry Butter Game.

  At the source document display (4810), a shared annotation associated with the source document and an indication of the portion of the source document associated with the shared annotation are received (4820). In this example, a single annotation is used for simplicity of explanation, but multiple annotations may be used. In that example, the multiple annotations are data, instructions or interactions used in the MMR as described herein. The annotations are stored in and retrieved from a shared documentation annotation server (SDA) such as 3755 shown in FIG. 37A in one embodiment. The SDA server 3755 is accessible via a network connection in one embodiment. The shared annotation extraction plug-in is, in this example, a plug-in 3720 as shown in FIG. 37A to facilitate this function. In another embodiment, annotations and instructions are received from the user. A user may create a shared annotation for a document that does not contain any annotations, or may add or modify an existing shared annotation to the document. For example, the user highlights (highlights) a portion of the source document and indicates an association with the shared annotation, which is prepared by the user in the various ways described.

  The modified document is then displayed in the browser (4830). The modified document includes a hot spot that corresponds to the portion of the source document specified in step 4820. Hotspots specify the location of shared annotations. According to one embodiment, the revision document is part of the revision file 3730 generated by the plug-in 3720 and is returned to the browser 3715. FIG. 49B shows a modified web page sample 4920 in a browser according to one embodiment of the present invention. Web page 4920 shows instructions for hotspot 4930 and associated annotation 4940, which in this example is a video clip. The instruction 4930 may be visually distinguished from the text of the remaining web page 4920, for example by highlighting. According to one embodiment, annotation 4040 is displayed when instruction 4930 is clicked or when the mouse is operated.

  In response to the print command, text coordinates and hot spots corresponding to the printed representation of the modified document are captured (4840). The details of coordinate acquisition may follow any of the methods described herein.

  Next, a printed page layout including hot spots is represented (4850). In one embodiment, the representation 4850 is to print the document. FIG. 49C shows a printed web page sample 4950 according to one embodiment of the present invention. The printed web page layout 4950 includes hot spots 4930 as indicated, but unlike web page 4920, the lines are interrupted in print layout 4950. In this example, the border of the hot spot 4930 is not visible in the printed layout 4950.

  In an optional final step, shared annotations are stored, for example, in data storage 3750 and indexed using relevance to hotspots 4930 in printed document 4950. The printed representation may also be stored locally. In one embodiment, the printing action causes a download and creation of a local copy (provides a trigger).

Image Processing Document Hot Spot FIG. 50A is a flow chart illustrating a method according to one embodiment of the present invention for adding a hot spot to an image processing document. The method allows hot spots to be added to a paper document after scanning, or allows hot spots to be added to a symbolic electronic document after being rendered for printing.

  First, the source document is converted into an image processing document (5010). In one embodiment, the source document is received at browser 3715 from source file 3710. The conversion (5010) may be performed by any method for generating a document to be subjected to feature extraction and generating a feature expression. According to one embodiment, a paper document is scanned and becomes an image processing document. In another embodiment, a representable page proof of an electronic document is rendered using a suitable application. For example, if the representable page proof is in Postscript format, a ghost script is used. FIG. 51A shows an example of a user interface 5105 showing a portion of a newspaper page 5110 scanned in one embodiment. The main window 5115 shows a large part of the newspaper page 5110, and the thumbnail 5120 shows which part of the page is displayed.

  Next, feature extraction is applied to the image processing document (5020) to create a feature representation. Any of the various feature extractions described in this application may be used for this purpose. In one embodiment, feature extraction is performed by the acquisition module 3735 described with reference to FIG. 37A. One or more hot spots 5125 are then added to the image processing document (5030). The hot spot may be predetermined or may need to be determined according to various examples. If the hotspot is already determined, the definition includes the number of pages, the location of the hotspot's bounding box on the page, the electronic data or interaction associated with the hotspot, etc. In one embodiment, the hotspot definition takes the form of a hotspot.xml file as shown in FIG.

  If the hot spot has not been determined, the end user may determine the hot spot. FIG. 50B shows a flowchart of a method according to one embodiment of the present invention for determining hot spots to be applied to an image processing document. First, a hot spot candidate is selected (5032). For example, in FIG. 51A, the end user uses a bounding box 5125 to select a portion of the document as a hot spot. Next, an optional step 5034 determines if the hotspot is unique for a given database. For example, there should be enough text to uniquely identify a hot spot within a patch surrounded by n ″ × n ″. A typical value for n is 2. If the photo spot is not sufficiently unique to the database, in one embodiment, the user is presented with options on how to handle the ambiguity. For example, the user interface may provide options such as selecting a larger area, allowing ambiguity but adding a description to the database. Other embodiments may use alternative methods of determining hot spots.

  Once the location of the hot spot is selected (5032), data or interaction is determined (5036) and added to the hot spot. FIG. 51B illustrates an example user interface that determines data or operations associated with a selected hot spot. For example, when the user selects the bounding box 5125, an edit box 5130 is displayed. Using the associated button, the user may cancel the operation (5135), simply save the bounding box 5125 (5140), or assign data or interactions to the hotspot (5145). ). If the user selects to assign data or interactions to the hotspot, an assignment box 5150 is displayed as shown in FIG. 51C. An assignment box 5150 allows the user to assign images 5155, various other media 5160 and web links 5165 to hotspots, which are distinguished by ID numbers. The user may then choose to save the hotspot definition (5175). Only one hot spot is shown for simplicity, but multiple hot spots may be used. FIG. 51D shows a user interface displaying hot spots in the document. In one embodiment, different colored bounding boxes correspond to different data and interaction types.

  In an optional step, the image processing document, hot spot definition, and feature representation are stored (5040), for example, in data store 3750.

  FIG. 52 illustrates a method according to one embodiment of the present invention using an MMR document 500 and an MMR system 100b.

  The method 5200 begins by obtaining a first document or a representation of the first document (5210). Specific methods for capturing the first document include the following (1) to (4): (1) Automatic text layout of the printed document within the operating system of the MMR computer 112 via the PD capture module 318. The first document is captured by automatically capturing; (2) the first document is captured by automatically capturing the text layout of the printed document within the printer driver 316 of the MMR computer 112; (3) a first document is captured, for example, by scanning a paper document via a scanned document scanner device 127 connected to the MMR computer 112; and (4) a file that is a representation of a printed document is automatically The first document is captured by transferring, uploading or downloading to the MMR computer 112 manually or manually. Although the capture step is described as capturing most or all of the printed document, it should be understood that the capture step 5210 may be performed on only a minimum portion of the printed document. Further, although the document is described in terms of capturing a single document, this step may be performed to capture multiple documents and create a library of first documents.

  Once the capture step 5210 is performed, the method 5200 performs indexing on the first document (5212). The indexing process makes it possible to identify the associated electronic representation of the document and the associated second media type and look for an input that matches the captured first document or part thereof. In this step in one embodiment, the document indexing is performed by a PD capture module 318 that generates a PD index 322. Specific examples of index processing include the following (1) to (6): (1) that the xy coordinates of the characters of the printed document are indexed, and (2) the xy coordinates of the words of the printed document. Is indexed, (3) the xy coordinates of the image or part of the image of the printed document are indexed, (4) OCR image processing is performed, and the character x and / or word x- the y coordinate is indexed accordingly, (5) feature extraction is performed from the image of the represented page, the xy coordinates of the feature are indexed, and (6) a symbolic version of the page Feature extraction is performed and the xy coordinates of the feature are indexed. The indexing process may include any or a group of the above index processes depending on the application of the present invention.

  The method 5200 captures a second document (5214). In this step 5214, the second document to be captured may be the entire document or only a portion (patch) of the second document. Specific examples of a method for capturing a second document include the following (1) to (8): (1) scanning a patch of text using one or more initial speed means 230 of the capture device 106; 2) Scanning a patch of text using one or more capture means 230 of the capture device 106 and then processing the image to determine the likelihood that the intended feature description is properly retrieved. For example, if the index is based on OCR, the system may determine whether the image contains multiple lines of text and whether the image is sharp enough for good OCR processing. If the determination is negative, another patch of text is scanned; (3) a machine readable identifier that identifies the document to be scanned (eg, International Standard Book Number (ISBN) or Universal Produce Code ( UPC)); (4) Enter data to identify the document or group of documents to be represented (for example, the 2003 edition of the Sports Photo Magazine), and then a text patch will be used in this method (1) Or (2) being scanned; (5) receiving an email with an attached second document; (6) receiving the second document by file transfer; (7) one of the capture devices 106; Scanning a portion of the image with the capture means 230 described above; and (8) inputting a second document with the input device 166.

  Once steps 5210 and 5214 are performed, the method performs a document or pattern match between the first document and the second document (5216). In one embodiment, this is done by performing a document fingerprint match of the first document and the second document. Document fingerprint verification is performed by querying the PD index 322 for the second media document. A specific example of document fingerprint matching is extracting features from the image captured in step 5214, constructing descriptors from those features, and looking for documents and patches that contain a certain percentage of descriptors. If the database stores a large number of documents, this pattern matching step is performed multiple times, once for each document, to see if any document in the library or database matches the second document. It should be understood that Alternatively, indexing step 5212 adds an index to document 5210, which represents a collection of documents and that the pattern matching step has been performed once.

  Finally, the method 5200 diplomatics an action based on the result of step 5216 and selectively based on user input (5218). In one embodiment, the method 5200 looks for a predetermined action, which is associated with a given document patch, for example the second media associated with the hotspot 506 that was found as matched in step 5216. Stored in 504. Specific examples of the predetermined actions include the following (1) to (6): (1) obtaining information from the document event database 320, the Internet, etc .; (2) MMR ready to receive the output of the system. Writing information to the location identified by system 100b; (3) looking for information; (4) displaying information on a client device such as capture device 106 and interactively interacting with the user; (5 ) Queue the actions and data determined in method step 5216 for later execution (user intervention may be optional); and (6) Immediately execute the data and actions determined in method step 5216 To do. Specific examples of the results of this method step include extraction of information, modified documents, execution of other actions (), input of commands sent to cable TV boxes such as set-top boxes 126, etc. Is coupled to a cable TV server (eg, service provider server 122) that streams video to a cable TV box. Once step 5218 is executed, the method 5200 is complete and ends.

  FIG. 53 shows a block diagram of a group of example business entities 5300 associated with the MMR system 100b according to one embodiment of the invention. Group of business entities 5300 are MMR service provider 5310, MMR consumer 5312, multimedia company 5314, printer user 5316, cellular telephone service provider 5318, hardware manufacturer 5320, hardware retailer 5322, financial institution 5324, credit card processing 5326, document publisher 5328, document printer 5330, fulfillment house 5332, cable TV provider 5334, service provider 5336, software provider 5338, advertising company 5340 and business network 5370.

  The MMR service provider 5310 is the owner and / or administrator of the MMR system 100 described with reference to FIGS. 1A-5 and 52. The MMR consumer 5312 represents any MMR user 110 previously described with reference to FIG. 1B.

  Multimedia company 5314 is some provider of digital multimedia products such as Block Blaster Inc. (Dallas, Texas), which provides digital movies and video games, and Sony Corporation of America, which provides digital music, movies and TV programs. (New York, New York).

  Printer user 5316 is any person or organization that uses some kind of printer to create printed paper documents. For example, the MMR consumer 5312 may be a printer user 5316 or a document printer 5330.

  Cellular telephone service provider 5318 is any cellular telephone service provider, such as Verizon Wireless (Bedminster, NJ), Singular Wireless (Atlanta, GA), T-Mobile USA (Beleb, WA) and Sprint Nextel (Reston, VA). Etc.

  Hardware manufacturer 5320 is any manufacturer of hardware devices, such as a printer, cellular phone or PDA. Specific examples of hardware manufacturers are, for example, Hewlett-Packard (Houston, Texas), Motoroline Corporation (Chicago, Illinois), and Sony Corporation of America (New York, NY). Hardware retailer 5322 is a retailer of anything from hardware, such as a printer, cellular phone or PDA retailer. Specific examples of hardware retailers include, but are not limited to, Radioshock Corporation (Fort Worth, TX), Circuit City Store Incorporated (Richmond, VA), Walmart (Bentonville, Arkansas), and Best Buy Corporation (Richfield, MN) ) Etc.

  Financial institution 5324 is some financial institution, such as some bank or credit association, and handles the management of bank accounts and the exchange of funds to and from other banks or financial institutions. Credit card processor 5326 is some credit card agency that manages credit card authentication and approves the process of purchase processing. Specific examples of credit card processors include, but are not limited to, ClickBank, which is Click Sales Incorporated (Boise, Idaho), Sharate Incorporated (Pralee, Minnesota), and SiCinau (CCNow) Incorporated (Eden, Minnesota). Prairie) service.

  Document publisher 5328 is any document publisher, including but not limited to Gregath Publisher (Wyandotte, Oklahoma), Prentice Hall (Upper Saddle River, New Jersey) and Pelican Publisher (Gretna, Louisiana). Etc. Document printer 5330 is any document printing company, including but not limited to PS Print LLC (Oakland, Calif.), Print Lizard Incorporated (Buffalo, NJ), and Mineo Incorporated (New York, NY). In another example, document publisher 5328 and / or document printer 5330 is any person (entity) that creates and distributes newspapers or magazines.

  A fulfillment house 5332 is any third party logistics warehouse specialized in order procurement as is well known. Specific examples of fulfillment houses include, but are not limited to, Corporate Disc Company (McHenry, Ill.), Order Motion Incorporated (New York, NY) and Shipwire.com (Los Angeles, CA).

  Cable TV provider 5334 is any cable TV service provider such as, but not limited to, Comcast Corporation (Philadelphia, PA) and Adelphia Communications (Greenwood Village, CO). Service provider 5336 represents some entity that provides some service.

  Software Provider 5338 is any software development company, including but not limited to Art and Logic Incorporated (Pasadena, Calif.), Jigsaw Data Corporation (San Mateo, Calif.), Data Mirror Corporation (New York, NY), and Databank IMX , LCC (Bertsville, Maryland), etc.

  Advertising company 5340 is some advertising company or agency, including but not limited to D & B Marketing (Elhurst, Illinois), Black Sheep Marketing (Boston, Massachusetts) and Gautam Direct Incorporated (New York, NY). is there.

  Business network 5370 represents any mechanism by which business relationships are established and / or supported.

  FIG. 54 illustrates a generalized business method 5400 prompted by utilizing the MMR system 100b according to one embodiment of the present invention. Method 5400 includes establishing a relationship between at least two entities, determining a potential business relationship, performing at least one transaction, and delivering / providing a product or service related to the transaction. Including.

First, a relationship is established between at least two business entities 5300 (5410). Business entities 5300 are organized into four broad categories, for example (1) MMR creator, (2) MMR distributor, (3) MMR users, and (4) others, and business entities are in more than one category. Applicable. According to this example, business entity 5300 is classified as follows:
MMR creators include MMR service provider 5310, multimedia company 5314, document issuer 5328, document printer 5330, software provider 5338, and advertising company 5340.

  MMR distributors are MMR service provider 5310, multimedia company 5314, cellular telephone service provider 5318, hardware manufacturer 5320, hardware retailer 5322, document issuer 5328, document printer 5330, fulfillment house 5332, cable TV Provider 5334, service provider 5336, advertising company 5340, and the like.

  The MMR users are the MMR consumer 5312, the printer user 5316, the document printing company 5330, and the like.

  ● Others include financial institutions 5324 and credit card processors 5326.

  For example, in the method steps, a business relationship is established between an MMR service provider 5310, an MMR creator, an MMR consumer 5312, an MMR user, a cellular telephone service provider, an MMR distributor, and a hardware retailer. In addition, hardware manufacturer 5320 has a business relationship with hardware retailer 5322, both of which are MMR distributors.

  Next, the method 5400 determines possible transactions among those having the relationship established in step 5410. In particular, various transactions are made between any two or more business entities 5300. Examples of transactions include: buying information; buying physical goods; buying services; buying bandwidth; buying electronic storage; advertising Purchasing advertising statistics; shipping goods; selling information; selling physical goods; selling services; selling bandwidth; selling electronic storage Selling advertisements; selling advertising statistics; lending / lease; gathering opinions / ratings / votings, etc.

  Once the method 5400 determines a possible transaction between each person, the MMR system 100 is used to reach an agreement on at least one transaction (5414). In particular, various actions may occur between any two or more business entities 5300 that result in a transaction result. Specific examples of actions include: buying information; receiving orders; clicking for more information; providing space; providing local / remote access. Hosting; shipping; generating transactions; storing private information; transmitting information to others; adding content; and podcasting.

  Once the method 5400 is agreed upon for the transaction, the MMR system 100 is used (5416) to distribute / provide the product or service associated with the transaction, for example, to the MMR consumer 5312. In particular, a variety of content is traded between any two or more business entities 5300 as a result of the business deal being reached at method step 5414. Specific examples of content include text, web links, software, still picture photos, video, audio and some combination thereof. In addition, various spreading means may be used between any two or more business entities 5300 to facilitate transactions. Specific examples of the spreading means include paper, a personal computer, a network computer, a capture device 106, a personal video device, a personal audio device, and some combination thereof.

  In addition to the invention described and claimed in the above embodiments, at least one form of one or more embodiments of the present invention provides a computer-implemented method for preparing mixed media documents. The method includes receiving (in an index table) an electronic description of features extracted from a paper document. The index table is for associating a paper document and the location of the features in the document with a mixed media document composed of digital media to be printed. Receiving a query term that captures a two-dimensional positional relationship between objects in the document to be searched, and a potential location candidate and at least one mixed media document according to the query term based on data from the index table The method follows the calculating step. In some such cases, the method includes storing additional features associated with the document to be searched. In some cases, such additional features include text information extraction, graphic information extraction, process execution, command execution, ordering, video extraction, sound extraction, information Including one or more actions including storing the document, creating a new document, printing the document, and / or displaying the document. In another particular case, receiving an electronic description of features extracted from a paper document (in an index table) includes receiving an electronic description of features extracted from a plurality of paper documents. In another particular case, calculating at least one mixed media document and location candidate includes calculating a ranked group of mixed media documents, pages, and location candidates based on data from the index table. . In another particular case, receiving a query term that captures a two-dimensional positional relationship between objects in a search target document includes receiving a group of laterally and vertically adjacent word pairs extracted from the search target document. including. In another particular case, the index table includes a reverse term index table, each unique term in the reverse term index table points to a list of records (records), and each record is a candidate area on a page in a mixed media document Is identified. In that case, computing at least one mixed media document and location candidate includes examining all records indexed with a key corresponding to the query term and identifying an area that best matches all query terms. Steps. If the identified region has a match score that meets the match criteria, the method may further include identifying corresponding mixed media documents and location candidates. In certain cases, the search target document is an image of a paper document or a patch of the paper document.

  At least one other form of one or more embodiments of the present invention includes machine-readable media encoded with instructions (eg, compact disc, diskette, server, memory stick, hard drive, ROM, RAM, etc.). Or any type of media suitable for storing electronic instructions) that, when executed by one or more processors, organizes information in a mixed media document system. And processing to access information is executed by the processor. The process may be similar to or different from the above method, for example.

  At least one other form of one or more embodiments of the present invention provides a database for preparing mixed media documents. The system includes an index table to receive an electronic description of the features extracted from the paper document and to associate the paper document with the location of the features in the document with the mixed media document that synthesizes the printed digital media Prepare for. The system includes an accumulator module, receives a query term that captures a two-dimensional positional relationship between objects in a search target document, and responds to the query term based on data from an index table. Compute mixed media documents and location candidates. In certain cases, the system includes a storage facility (eg, a relational database) and stores additional features associated with the document being searched. Text information extraction, graphic information extraction, process execution, command execution, arranging in a certain order, extracting video, extracting sound, storing information, creating new documents, It includes one or more actions including printing the document and / or displaying the document. In another particular case, the index table can receive an electronic description of features extracted from multiple paper documents. In another particular case, the paper document includes a plurality of pages, and the index table is constructed to identify the mixed media document, pages, and xy location attributes within those pages. In another particular case, calculating at least the mixed media document and location candidates executed in the accumulator module calculates a ranked group of location candidates, pages and mixed media documents based on data from the index table. Including doing. In another particular case, receiving a query term (performed by the accumulator module) that captures a two-dimensional positional relationship between objects in the search target document is horizontally and vertically extracted from the search target document. Receiving adjacent word pairs. In another particular case, the index table includes a reverse term index table, each unique term in the reverse term index table points to a list of records (records), and each record is a candidate area on a page in a mixed media document Is identified. In such a case, calculating at least one mixed media document and location candidate is to examine all the records indexed by the key corresponding to the query term and to determine the region that best matches all the query terms. Identifying. If the identified region has a match score that meets the match criteria, the accumulator module may identify the corresponding mixed media document and location candidate. In another such case, the index table further includes a document index table that includes relevant information for each mixed media document, the relevant information including at least one of print resolution, print date, paper size, shadow file name, and page image location. Including one. In another particular case, the descriptions received in the index table are calculated in a feature extraction module that associates the extracted features with the internal document location data for those features. In certain cases, the document to be searched is an image of a paper document or a patch of that paper document. The functions of the system can be implemented in conjunction with various means such as software (eg, executable instructions encoded on one or more computer-readable media), hardware (eg, Gate level logic or one or more ASICs), firmware (eg, one or more microcontrollers with I / O functions and embedded routines that perform the functions described herein), or some combination thereof is there. The database system can be implemented with a mixed media reality (MMR) system as described herein, which can be implemented, for example, on another server, on a computer system, on a portable device, or some combination thereof. It has a function to be performed.

  In a particular embodiment, the MMR system includes a content-based extraction database along with an index table, which is a method that allows searching using a text-based index. Represents a dimensional geometric positional relationship. A ranked group of documents, pages and location candidates can be calculated under a given data from the index table. This technique efficiently converts the features detected in the image patch into text terms (or other searchable features), such as text terms themselves and the geometric positional relationship between them. Express both. The storage means can be used to store additional features for each document image patch.

  The algorithm presented here is not uniquely associated with any particular computer or other device. Various general purpose and / or dedicated systems may be programmed or constructed according to embodiments of the invention. As will be apparent from this disclosure, various programming languages and / or structures can be used to implement such various systems. Further, embodiments of the invention may operate on or function within an information system or network. For example, the present invention may operate on a stand-alone multifunction printer or may operate on a networked printer with features that vary depending on the configuration. With the minimum of all the functions described here, the present invention can also work with any information system.

  The above description of embodiments of the invention is intended for purposes of illustration and description. It is not intended to limit the invention to the precise form disclosed or to limit the disclosure. Many modifications and variations are possible from the above teachings. It is intended that the scope of the invention be limited not by this detailed description, but only by the claims of this application. As will be appreciated by those skilled in the art, the present invention may be incorporated in other forms without departing from the spirit or essential characteristics of the invention. Similarly, specific partitioning schemes and naming of modules, routines, features, attributes, methods and other forms are neither essential nor important. The present invention or means (mechanism) for realizing the features of the present invention may have various names, classification methods and / or formats. Further, as will be apparent to those skilled in the relevant arts, other aspects of the invention, such as modules, routines, features, attributes, methods, etc., are software, hardware, middleware, or some combination thereof. It is feasible. The component of the present invention (for example, it is a module) is realized as software, and the component can be realized as a stand-alone program, as a statically or dynamically linked library, as a kernel loadable module, as a device driver And / or implemented in any manner now known or future known to those skilled in the art of computer programming. Further, the present invention is not limited to being implemented in any particular programming language, and is not limited to any particular operating system or environment. Accordingly, the disclosure of the present invention is illustrative and is not intended to limit the scope of the invention as set forth in the appended claims.

  This application is a U.S. priority application 60 / 710,767 filed on August 23, 2005, a U.S. application 60 / 792,912 filed on April 17, 2006, and a U.S. application filed on July 31, 2006. No. 461,147, based on US application Ser. No. 11 / 461,164, filed Jul. 31, 2006, the entire contents of which are incorporated herein by reference.

FIG. 3 shows a functional block diagram of a mixed media reality (MMR) system constructed in accordance with one embodiment of the present invention. FIG. 4 shows a functional block diagram of an MMR system constructed in accordance with another embodiment of the present invention. 1 shows a capture device according to one embodiment of the present invention. 1 shows a capture device according to one embodiment of the present invention. 1 shows a capture device according to one embodiment of the present invention. 1 shows a capture device according to one embodiment of the present invention. FIG. 3 is a functional block diagram of a capture device constructed in accordance with an embodiment of the present invention. 1 shows a functional block diagram of an MMR computer constructed in accordance with one embodiment of the present invention. FIG. 3 is a diagram illustrating software component groups included in MMR software appropriately constructed according to an embodiment of the present invention. FIG. 6 shows a diagram representing an example of an MMR document constructed in accordance with one embodiment of the present invention. It is a figure for demonstrating the fingerprint collation method of the document by one Example of this invention. 1 shows a diagram representing a document fingerprint verification system constructed in accordance with one embodiment of the present invention. FIG. FIG. 6 shows a diagram representing a flowchart for distinguishing text / non-text according to one embodiment of the present invention. FIG. 4 shows a diagram illustrating a specific example of distinguishing text / non-text according to an embodiment of the present invention. FIG. 5 shows a diagram representing a flow chart for estimating the point size of text in an image patch according to one embodiment of the present invention. FIG. 6 is a diagram for explaining a fingerprint collation method for a document according to another embodiment of the present invention. FIG. 6 is a diagram for explaining a fingerprint collation method for a document according to another embodiment of the present invention. FIG. 6 is a diagram illustrating an example of interactive image analysis according to an embodiment of the present invention. FIG. 6 is a diagram for explaining a fingerprint collation method for a document according to another embodiment of the present invention. It is a figure which shows the example of word boundary box determination by one Example of this invention. It is a figure which shows the feature extraction method by one Example of this invention. It is a figure which shows the feature extraction method by another Example of this invention. It is a figure which shows the feature extraction method by another Example of this invention. It is a figure which shows the feature extraction method by another Example of this invention. FIG. 6 is a diagram for explaining a fingerprint collation method for a document according to another embodiment of the present invention. FIG. 10 is a diagram illustrating an example of multi-classifier feature extraction for document fingerprint matching according to an embodiment of the present invention. It is a figure which shows the fingerprint collation example of the document by one Example of this invention. It is a figure which shows the fingerprint collation example of the document by one Example of this invention. It is a figure which shows the fingerprint collation example of the document by another Example of this invention. 3 shows a flowchart of database driven feedback according to an embodiment of the present invention. It is a figure which shows the fingerprint collation example of the document by another Example of this invention. 2 shows a flowchart of database driven classification according to one embodiment of the present invention. It is a figure which shows the fingerprint collation example of the document by another Example of this invention. 6 shows a flowchart of database driven multiple classification according to one embodiment of the present invention. It is a figure which shows the fingerprint collation example of the document by another Example of this invention. It is a figure which shows the fingerprint collation example of the document by another Example of this invention. It is a figure which shows the fingerprint collation example of the document by another Example of this invention. 6 shows a flowchart of multi-tier recognition according to an embodiment of the present invention. 1 shows a functional block diagram of an MMR database system constructed according to one embodiment of the present invention. FIG. FIG. 5 is a diagram illustrating an example of extracting an MMR feature of an OCR base method according to an embodiment of the present invention. It is a figure which shows the index table organization example by one Example of this invention. FIG. 6 is a diagram illustrating a method for generating an MMR index table according to an embodiment of the present invention. FIG. 6 illustrates a method according to an embodiment of the present invention for calculating a ranked group of documents, pages and location hypotheses for a search target document. FIG. 4 shows a functional block diagram of an MMR component constructed in accordance with another embodiment of the present invention. FIG. 3 illustrates a group of software components included in MMR printing software according to one embodiment of the present invention. 2 is a flowchart illustrating a method according to one embodiment of the present invention for embedding hot spots in a document. It is a figure which shows the example HTML file by one Example of this invention. It is a figure which shows the example of the markup version of the HTML file of FIG. 39A. It is a figure which shows the example of an HTML file of FIG. 39A displayed on a browser by one Example of this invention. FIG. 40B is a diagram illustrating an example of a printed version of the HTML file of FIG. 40A according to one embodiment of the present invention. FIG. 6 is a diagram illustrating a symbolic hot spot description according to an embodiment of the present invention. FIG. 39B is a diagram showing an example page_desc.xml file for the HTML file of FIG. 39A according to one embodiment of the present invention. FIG. 39B is a diagram showing an example page_desc.xml file for the HTML file of FIG. 39A according to one embodiment of the present invention. FIG. 43 is a diagram showing a hotspot.xml file corresponding to FIGS. 41, 42A and 42B according to one embodiment of the present invention. FIG. 4 shows a flow chart of a process according to an embodiment of the invention used in a forwarding DLL. 4 is a flowchart illustrating a method according to an embodiment of the present invention for converting a character corresponding to a hot spot in a document. It is a figure which shows the electronic version example of the document by one Example of this invention. FIG. 5 is a diagram illustrating an example of a printed modified document according to an embodiment of the present invention. 6 shows a flowchart of a document annotation method shared by one embodiment of the present invention. FIG. 3 shows a sample source web page in a browser according to one embodiment of the present invention. FIG. 4 is a diagram illustrating a modified web page sample in a browser according to an embodiment of the present invention. FIG. 4 illustrates a printed web page sample according to one embodiment of the present invention. 6 is a flowchart illustrating a method according to one embodiment of the present invention for adding a hot spot to an image processing document. 6 is a flowchart illustrating a method according to one embodiment of the present invention for determining hot spots to add to an image document. FIG. 6 is a diagram illustrating an example user interface showing a portion of a scanned newspaper page in one embodiment. It is a figure which shows the example of a user interface which determines the data or operation linked | related with the selected hot spot. FIG. 51B illustrates the example user interface of FIG. 51B including an insertion box according to one embodiment of the present invention. FIG. 5 is a diagram illustrating a user interface displaying hot spots in a document according to an embodiment of the present invention. 4 is a flowchart illustrating a method according to an embodiment of the present invention using an MMR document and an MMR system. FIG. 4 shows a block diagram of an example group of business entities associated with an MMR system according to one embodiment of the invention. Fig. 4 shows a flow chart of a generalized business method that is facilitated by utilizing an MMR system according to one embodiment of the present invention.

Explanation of symbols

100a, 100b MMR (mixed media reality) system
102 MMR processor
104 Communication means
106 Trapping device
108 MMR software
110 users
112 MMR computer
114 Network media server
116 Printer
118 printed documents
120 office portal
122 Service Provider Server
124 electronic display
126 set top box
127 Document scanner
160 Base media storage
162 MMR media storage
164 Output device
166 Input device
168 Portable input device
170 Portable output device
210 processor
212 display
214 keypad
216 storage devices
218 Wireless communication link
220 Wired communication link
222 MMR soft air
224 Capture Device User Interface (UI)
226 Document Fingerprint Verification Module
228 Third Party Software Module
230 Trapping means
232 camcorder
234 Still camera
236 Voice Recorder
238 Electronic Highlighter (Electronic Highlighter)
240 laser
242 GPS device
244 RFID reader
310 source files
312 First Source Document (SD) Browser
314 Second SD Browser
316 Printer device
318 Print Document (PD) Capture Module
320 Document Event Database
322 PD index
324 Event Capture Module
326 Document Analysis Module
328 Multimedia (MM) clip browser / editor module
330 MM printer driver
332 Document Video Analysis (DVP) Printing System
334 video paper
410 Multimedia annotation software
412 Text Content Based Extraction Component
414 Image Content Based Extraction Component
416 Steganographic correction component
418 Paper reading history log
420 Online reading history log
422 Compatibility Document Confirmation Component
424 Real-time notification component
426 Multimedia Acquisition Component
428 Desktop Video Reminder Component
430 Web Page Reminder Component
432 Physical history log
434 Completed Form Confirmation Component
436 Time Propagation Component
438 Location component
440 PC initialization component
442 Document Authoring Component
444 Capture Device Authoring Component
446 Unconscious (automatic) upload component
448 Document Version Acquisition Component
450 PC Document Metadata Component
452 Capture Device User Interface (UI) Component
454 Domain specific components
610 Document fingerprint verification system
712 quality evaluation module
3400 database system
3402 MMR feature extraction module
3404 MMR index table module
3406 Evidence storage module
3408 relational database
3414 Document Conversion Application Module
3416 Sub-image extraction module
3418 Feedback-oriented feature search module
3705 computer
3710 source files
3715 Browser
3720 plug-in
3725 Symbolic hotspot description
3730 Modified file
3735 acquisition module
3740 page_desc.xml
3745 hotspot.xml
3750 data store
3755 SDA server
3760 MMR printing software
3765 conversion module
3768 embedded modules
3770 analysis module
3775 conversion module
3778 Feature Extraction Module
3780 annotation module
3785 Hotspot module
3790 representation / display module
3795 storage module
5300 Business entity
5310 MMR Service Provider
5312 MMR Consumer
5314 Multimedia companies
5316 Printer user
5318 Cellular telephone service provider
5320 Hardware manufacturer
5322 Hardware retailers
5324 Financial institutions
5326 Credit card processor
5328 Document publishers
5330 Document printer
5332 fulfillment house
5334 Cable TV Provider
5336 Service Provider
5338 Software Provider
5340 Advertising companies
5370 Business Network

Claims (35)

A computer-implemented method for organizing and accessing information in a mixed media document system comprising:
Generating an electronic representation of the paper document;
Identifying features on the paper document and capturing a two-dimensional form of the paper document;
Identifying the location of the feature;
Indexing the features by the location and generating an index table;
Having a method.   The method of claim 1, further comprising a preliminary step of receiving the paper document.   The method of claim 1, further comprising storing one or more features associated with at least one of the features.   The one or more features include: extraction of text information; extraction of graphic information; execution of processes; execution of commands; arranging in a certain order; extracting video; extracting sound; storing information; 4. The method of claim 3, including one or more actions including at least one of creating a new document, printing the document, and displaying the document.   The method of claim 1, wherein identifying features on the paper document and capturing a two-dimensional form of the paper document includes identifying horizontally aligned objects and vertically aligned objects.   The method of claim 1, wherein identifying features on the paper document and capturing a two-dimensional form of the paper document includes identifying a horizontal word pair and a vertical word pair.   The method of claim 1, wherein the step of generating an electronic representation of the paper document is performed during a scanning or printing process.   The step of identifying features on the paper document and capturing the two-dimensional form of the paper document is to examine the amount of vertical overlap between two successive sequences, thereby converting the sequence of text into logical lines. The method of claim 1 including the step of grouping into: Receiving one or more query terms and capturing a two-dimensional positional relationship between objects in the search target document;
Calculating at least one mixed media document and location candidates that may be responsive to the query terms based on data from the index table;
The method of claim 1 further comprising: Prior to receiving one or more query terms and capturing a two-dimensional positional relationship between objects in the document to be searched,
Receiving the search object document; and
Creating at least a patch image of the document to be searched;
Generating one or more query terms based on the image;
10. A method according to claim 9, comprising:   The method of claim 10, wherein generating one or more query terms based on the image includes generating horizontal and vertical word pairs extracted from the image. Calculating at least one mixed media document and location candidates;
Identifying a stored page that tends to best match the patch of the search target document;
Calculating a location closest to the center of the patch in the page;
The method of claim 10 comprising: Each word pair is associated with an inverse document frequency function,
Identifying a stored page that tends to best match at least a patch of the document to be searched;
Adding the inverse document frequency function of each word pair to an accumulator indexed on the document page in which the word pair appears;
Outputting a document page that matches the patch with a corresponding document page according to a maximum value in the accumulator that exceeds a threshold;
The method according to claim 12, comprising: Calculating the location in the page closest to the center of the patch;
Weighting each cell in a zone around each word pair, wherein the weight of each cell is normalized between the inverse document frequency of the word pair and the center of the cell and the zone A step determined by the product of the geometric distance;
Finding the corresponding Accum array of the cell accumulator with the maximum value;
Notifying the cell as the location of the patch according to a maximum value exceeding a threshold;
14. The method of claim 13, comprising: Calculating at least one mixed media document and location candidates;
Looking up each of the one or more query terms in an index table for searching one or more locations associated with each query term;
Identifying, for each identified location, one or more candidate regions that include the location;
10. A method according to claim 9, comprising: Calculating at least one mixed media document and location candidates;
Identifying one of the one or more candidate regions that best matches all of the one or more query terms;
Determining that the one of the one or more candidate regions meets a predetermined match criterion, and determining the region as matching the search target document;
The method according to claim 15, comprising: A computer readable medium encoded with instructions, wherein the instructions, when executed on one or more processors, include a process for organizing and accessing information in a mixed media document system. Of the processor, the process
Generating an electronic representation of a paper document,
Identifying features on the paper document and capturing a two-dimensional form of the paper document;
Identifying the location of the feature;
Indexing the features by the location and generating an index table;
A computer readable medium including The process is
The computer-readable medium of claim 17, further comprising storing one or more features associated with at least one of the features.   18. The computer readable medium of claim 17, wherein identifying features on the paper document and capturing a two-dimensional form of the paper document includes identifying horizontal and vertical word pairs. The process is
Receiving one or more query terms, capturing a two-dimensional positional relationship between objects in the document to be searched, and at least responding to the query terms based on data from the index table Calculating one mixed media document and location candidates;
The computer-readable medium of claim 17 further comprising: A computer-implemented method for accessing information in a mixed media document system, comprising:
Receiving one or more query terms and capturing a two-dimensional positional relationship between objects in the search target document;
Calculating at least one mixed media document and location candidate that may respond to the query terms based on data from an index table;
And wherein the index table indexes document features and feature locations of mixed media documents. Prior to receiving one or more query terms and capturing a two-dimensional positional relationship between objects in the document to be searched,
Receiving the search object document; and
Creating at least a patch image of the document to be searched;
Generating one or more query terms based on the image;
The method of claim 21, comprising:   23. The method of claim 22, wherein generating one or more query phrases based on the image includes generating horizontal and vertical word pairs extracted from the image. Calculating at least one mixed media document and location candidates;
Identifying a stored page that tends to best match the patch of the search target document;
Calculating a location in the page closest to the center of the patch;
24. The method of claim 23, comprising: Each word pair is associated with an inverse document frequency function,
Identifying a stored page that tends to best match at least a patch of the search target document;
Adding the inverse document frequency function of each word pair to an accumulator indexed on the document page in which the word pair appears;
Outputting a corresponding document page as the patch niche in response to a maximum value in the accumulator exceeding a threshold;
25. The method of claim 24, comprising: Calculating the location in the page closest to the center of the patch;
Adding a weight to each cell in a zone around each word pair, wherein the weight of each cell is normalized between the inverse document frequency of the word pair and the center of the cell and the zone A step determined by the product of the geometric distance;
Finding the corresponding Accum array of the cell accumulator with the maximum value;
Reporting the cell as the location of the patch in response to a maximum value exceeding a threshold;
26. The method of claim 25, comprising: Calculating at least one mixed media document and location candidates;
Looking up each of the one or more query terms in an index table for searching one or more locations associated with each query term;
Identifying, for each identified location, one or more candidate regions that include the location;
The method of claim 21 comprising: Calculating at least one mixed media document and location candidates;
Identifying one of the one or more candidate regions that best matches all of the one or more query terms;
Confirming that the one of the one or more candidate regions meets a predetermined match criterion, and determining the region as matching the search target document;
28. The method of claim 27, comprising: A machine-readable medium encoded with instructions, where the instructions, when executed on one or more processors, cause the processor to execute a process for accessing information in a mixed media document system, the process comprising: ,
Receiving one or more query terms and capturing a two-dimensional positional relationship between objects in the search target document;
Calculating at least one mixed media document and location candidate that may respond to the query terms based on data from an index table;
Wherein the index table indexes document features and feature locations of mixed media documents. Prior to receiving one or more query terms and capturing a two-dimensional positional relationship between objects in the document to be searched,
Receiving the search object document; and
Creating at least a patch image of the document to be searched;
Generating one or more query terms based on the image;
30. The machine readable medium of claim 29, comprising:   31. The machine readable medium of claim 30, wherein generating one or more query terms based on the image includes generating horizontal and vertical word pairs extracted from the image. Calculating at least one mixed media document and location candidates;
Identifying a stored page that tends to best match the patch of the search target document;
Calculating a location in the page closest to the center of the patch;
32. The machine readable medium of claim 31, wherein: Each word pair is associated with an inverse document frequency function,
Identifying a stored page that tends to best match at least a patch of the search target document;
Adding the inverse document frequency function of each word pair to an accumulator indexed on the document page in which the word pair appears;
Outputting a corresponding document page as the patch niche in response to a maximum value in the accumulator exceeding a threshold;
35. The machine-readable medium of claim 32, comprising: Calculating the location in the page closest to the center of the patch;
Adding a weight to each cell in a zone around each word pair, wherein the weight of each cell is normalized between the inverse document frequency of the word pair and the center of the cell and the zone A step determined by the product of the geometric distance;
Finding the corresponding Accum array of the cell accumulator with the maximum value;
Reporting the cell as the location of the patch in response to a maximum value exceeding a threshold;
34. The machine readable medium of claim 33, comprising: Calculating at least one mixed media document and location candidates;
Looking up each of the one or more query terms in an index table for searching one or more locations associated with each query term;
Identifying, for each identified location, one or more candidate regions that include the location;
Identifying one of the one or more candidate regions that best matches all of the one or more query terms;
Confirming that the one of the one or more candidate regions meets a predetermined match criterion, and determining the region as matching the search target document;
30. The machine readable medium of claim 29, comprising:

Images