JP4576211B2 - Document information retrieval system - Google Patents

Description

  The present invention relates to a system for searching related information that is artificially embedded in a character string in a document, and creates a search DB corresponding to the character string in the document and the related information in advance, Relevant information registered in the search DB from the plurality of character strings by recognizing a character string by character recognition from an image obtained by capturing a part of a document captured by a camera built in a mobile phone or the like Is to search.

Conventionally, digital watermarking has been known as a method for embedding information in a printed photograph or document. For example, as an example of embedding a digital watermark in a binarized image such as a document, there is Japanese Patent Laid-Open No. 2004-289883 (Patent Document 1).
Conventionally, various techniques for recognizing characters in a character string included in an image taken by a portable terminal such as a mobile phone equipped with a camera and translating the character text as a recognition result have been proposed. For example, Japanese Patent Application Laid-Open No. 09-138802 (Patent Document 2) has a character recognition function and a translation function inside a mobile terminal, and uses these functions to recognize a character string in an image taken by a camera. A translation system for translation processing is disclosed.

  Also, in the previous application of the same applicant, Japanese Patent Application No. 2004-227610 and Japanese Patent Application No. 2004-232891, the shooting position in the document is specified from the arrangement and positional relationship of a plurality of words in the image shot with the portable terminal. Then, there is an invention related to a document translation system that displays a translation result related to a photographed word and a link destination URL set for a position in a document on a browser.

Japanese Patent Application Laid-Open No. 2004-289883

JP 09-138802 A

In Patent Document 1, it is necessary to print and distribute a document in which information is embedded in advance. For this reason, digital watermarks cannot be applied to printed books.
Further, in Patent Document 2, a specific place in a document is photographed, a character image in an image photographed by character recognition is converted into a character string including a word or sentence, and the word or sentence is output. A translation system is disclosed for translating. If the recognition range is a sentence, it is necessary to increase the resolution of the camera because the shooting range is widened, or to recognize the character string from multiple captured images, reconstruct the subdivided character string, and reproduce the sentence there were. Also, when translating words, the translation may differ depending on the position of the word in the document.

  Moreover, since character recognition has a possibility of misreading, it is necessary to improve a recognition rate. FIG. 32 shows an example of the flow of character recognition processing. The image binarization unit 51 creates a binary image in which a background and characters are separated from an image including characters. The character line cutout unit 52 determines a character string area for one line from the binary image. The character cutout unit 53 determines a character area of one character from the character string area for one line. The feature extraction unit 54 calculates a character feature from a character region of one character. The identification processing unit 55 compares the character features with the character features registered in advance for each character, and outputs the character candidates in the order in which the features are similar. The word collating unit 56 outputs the word closest to the word registered in the word collation dictionary from the character candidates for each continuous character region. In the image binarization unit 51, the printed document has a wide range of conditions such as color printing and halftone printing, and may not be binarized correctly by one type of binarization method. The character line segmentation unit 52 and the character segmentation unit 53 need to predict the character line and the character area from the image before recognizing the character, but there are various types of documents such as ruled lines, line spacing, and character spacing. Misidentification may occur. In the identification processing unit 55, there are similar characters (for example, very similar characters such as date, white, eyes, etc.), and character candidates may not be output correctly due to an adverse effect due to image noise or blurring. The word collation unit 56 needs to register all words in the word collation dictionary to improve the correct reading rate. However, a large memory is required to register words of all published documents. On mobile terminals such as mobile phones with restrictions, all words cannot be registered in the word matching dictionary.

  An object of the present invention is to specify a location in a document where a character string included in an image obtained by photographing a part of a document with a camera using a portable information terminal with a built-in camera, and to the location. An object of the present invention is to provide a document information retrieval system for acquiring associated information.

Another object of the present invention is to provide a document information search system that automates the creation of data for specifying the location of characters and simplifies the creation of information data associated with a document. .
Another object of the present invention is to create a character recognition parameter by extracting document features from a document file that defines the document structure for each document, and set the character recognition parameter according to the document to be searched. An object of the present invention is to provide a document information search system that can improve the character recognition rate by setting.

The present invention relates to a character included in an image photographed by an image photographing unit using a portable information terminal having an image photographing unit, an input unit operated by a user to input information, and a display unit. In a document information retrieval system that obtains information and displays the information on a display unit,
A document information registration unit that creates related information in which information related to document components is associated; a coordinate registration unit that creates position information of a character string; and a storage device that holds the related information and the position information. Is. In addition, by providing a document selection unit, related information registered in the storage device for each document and information indicating the document are stored so that the position information can be selected and the search unit can retrieve the information.

  In the present invention, a server device includes the storage device and the search unit, and the portable information terminal transmits a character string obtained from the character recognition unit to the server via a network. The information obtained as a result of searching for related information by the search unit is transmitted to the portable information terminal via the network. The character recognition unit may be provided in a server device.

  The document information registration unit of the present invention inputs a document file that defines the document structure, analyzes the document structure of the document file, that is, character, character type, character coordinates, character size, etc. The information is divided into elements (chapter, column, sentence, line, heading, etc.), and related information in which information related to the constituent element unit is associated is created. For example, the document file 15 shown in FIG. 5 has a PDF document structure. A PDF file has a plurality of objects in which documents are hierarchized, such as a page object and a text object. The text object describes the typeface, font, character width specification, start position on the page, character code, and the spacing for each character. By using these pieces of information, it is possible to easily estimate the position of the character string.

  More preferably, the constituent elements of the document are hierarchized, the related information is set from the upper hierarchy, and the related information is searched from the lower hierarchy when searching. More preferably, the constituent elements of the document are decomposed into morphemes, and related information in which information related to the morpheme unit is associated is created. Still more preferably, the morpheme has an editing unit in which a registrant edits a morpheme break.

  According to the present invention, since a unit for setting related information such as a line, a sentence, and a word is created from a character string and a character position of the document, a registrant does not need to register a line, a sentence, a word, or the like from the document. Become.

  The document information registration system according to the embodiment of the present invention includes a document registration device, a server device, and a mobile terminal device. As the related information set in the document to be searched, a URL (Uniform Resource Locator) is set for each line and word, and the information is displayed on the browser using the URL. Also, a PDF (Portable Document Format) file will be described as an example of a document file that defines the structure of a document to be input as a numerical value.

  An embodiment of the present invention will be described with reference to FIG. FIG. 1 is an example of an overall configuration diagram of the present embodiment. A document registrant operates the document registration apparatus 10 in advance, and the document input apparatus 41 inputs document data for a document for registering related information. The analysis device 42 disassembles the input document file into constituent elements constituting the document such as lines and words, and displays the disassembled constituent elements on the display device 40. The document registrant inputs information related to the component from the input device 45. The search DB including the component and related information corresponding to the component is stored in the storage device 44 by the registration device 43. The search DB is transmitted from the communication device 46 of the document registration device 10 to the communication device 33 of the server device 320 via the network 36 and stored in the storage device 32 of the server device 320.

  The mobile terminal device 300 displays a list of documents registered with related information on the display device 22. A user of the portable terminal device 300 selects a document to be photographed for searching for related information by the operation of the input device 20 using the document selection device 21 and photographs a part of the printed document by the photographing device 23. The character recognition device 24 recognizes a word of interest and a plurality of surrounding words from the captured image. The mobile terminal device 300 is an information terminal having an imaging device such as a mobile phone or a PDA.

  In the server device 320, the position search device 34 specifies the position in the document of the attention word photographed by the mobile terminal device 300 from the search DB stored in the storage device 32, and the information search device 31 corresponds to the attention word. Related information is searched from the search DB. The display device 22 of the mobile terminal device 300 displays related information obtained by the information search device 31. Data communication between the mobile terminal device 300 and the server device 320 is performed by connecting the communication device 25 and the communication device 30 via the network 26, and the attention word recognized by the mobile terminal device 300 and the relationship searched by the server device 320. Transfer information.

  Next, details of the document registration apparatus 10 in FIG. 1 according to the embodiment of the present invention will be described in detail with reference to FIG. 2 will be described below using the flowchart of FIG. 3 and the reference numerals of FIGS. 5 and 8 to 12 as necessary. FIG. 2 is an example of a block diagram showing the operation of the document registration apparatus of this embodiment. FIG. 3 is a flowchart showing the operation of the document registration apparatus of this embodiment.

  The correspondence between the blocks in FIGS. 1 and 2 will be described. The document input device 41 corresponds to the document registration unit 101 and the document input unit 100. The analysis device 42 corresponds to the attribute extraction unit 102, document coordinate extraction unit 106, document extraction unit 103, morpheme analysis unit 104, word editing unit 105, word list creation unit 107, and collation dictionary creation unit 109. The registration device 43 corresponds to the link destination registration unit 111. The storage device 43 corresponds to the document information 120, document attribute information 121, word collation dictionary 122, word information 123, character information 126, line link information 124, and word link information 125.

  In the document registration unit 101, the registrant inputs the document title 201 from the document registration unit 101 (130). The registrant inputs the document file 15 through the document input unit 100 (131). The attribute extraction unit 102 analyzes the document structure of the document file 15 and extracts the document page number 202, character background, character direction, and used character type (132). The page number 202 and the document registration unit 101 The title 201 input by the above and the automatically generated document number 200 are registered in the document information 120 (133). Further, the binarization mode 70, the character direction 71, and the character code 72 are created from the extracted character background, character direction, and character type used, and are registered in the document attribute information 121 (150).

  The character coordinate extraction unit 106 extracts the position coordinates of the characters in the page (160), and outputs position information such as the page number 210, the row number 211, and the column number 233 for each character. The character extraction unit 103 extracts characters from the document file 15 (135), reconstructs a character string on the same line from the page number 210 and line number 211 of each character set by the character coordinate extraction unit 106, and character information 126 (139).

  The morpheme analysis unit 104 parses the character string from the character information 126 into a plurality of parts of speech constituting the line by performing morphological analysis on the line unit (136). Unnecessary parts of speech such as conjunctions are removed, and parts of speech that can be used as words such as proper nouns are selected and output (137). The word editing unit 105 combines the character strings constituting a plurality of continuous parts of speech, or the characters constituting the part of speech, which the registrant performs as necessary on the character strings constituting the part of speech output by the morphological analysis unit 104. Editing such as column division is accepted and a word 230 is created (138). The word list creation unit 107 uses the word 230 output from the word editing unit 105 and the page number 210, the line number 211, and the column number 233 output from the character coordinate extraction unit 106 of the first character of the word 230. (140).

  The collation dictionary creating unit 109 creates the word collation dictionary 122 from all the words 230 registered in the word information 123 (171). The link destination registration unit 111 inputs from the registrant information on the link destination URL corresponding to the character data 212-1 to b of the character information 126 and the word data 234-1 to c registered in the word information 123. The line link information 124 and the word link information 125 are created (143).

Hereinafter, the document information 120, the character information 126, the word information 123, the line link information 124, and the word link information 125 created by the document registration apparatus 10 will be described with reference to FIGS.
The document information 120 will be described with reference to FIG. In FIG. 8, the document information 120 is composed of document data 203-1 to a, and the document data 203-1 to a is composed of a document number 200, a title 201, and a page number 202 for each document. The document number 200 is a number automatically set for each registered document, the title 201 is the title of the document, and the page number 202 indicates the total number of pages constituting the document.

  Next, the character information 126 will be described with reference to FIG. In FIG. 9, the character information 126 is composed of character data 212-1 to 212-b, and the character data 212-1 to b are printed with a character string 213 and a character string 213 for each line unit in the document. , Page number 210 and line number 211.

  Next, the word information 123 will be described with reference to FIG. In FIG. 10, word information 123 includes word data 234-1 to 234-c, and the word data 234-1 to 234-c includes a word 230, a page number 210 indicating the position of the word 230 in the printed document, a line It consists of number 211 and column number 233. The word 230 is a word created by the word list creation unit 107. The column number 233 is a position where the first character of the word 230 is counted from the left character of the character string 213 in the character string 213 in the document information 126 including the word 230.

  Next, the row link information 124 will be described with reference to FIG. In FIG. 11, the line link information 124 is composed of line link data 222-1 to 22-b, and the line link data 222-1 to 222-b is a page number 210 and a line number set for all the lines in the document. 211 and a link destination URL 223.

  Next, the word link information 125 will be described with reference to FIG. In FIG. 12, the word link information 125 includes word link data 235-1 to 235-1e, and the word link data 235-1 to 235-1 to the word 230, the link destination URL 223 set in the word 230, and the printed document of the word 230 It consists of page number 210, row number 211, and column number 233 indicating the position inside.

  Hereinafter, a procedure for extracting the character information 126 from the document file 15 taking a PDF file as an example will be described in detail with reference to FIGS. FIG. 5 is an example of a document file input according to the present embodiment, and FIG. 28 is a flowchart showing a procedure for creating character information 126 according to the present embodiment.

  The meaning of each representative line in the PDF file example of FIG. 5 will be described using line number 16. To make the explanation easier to understand, the embedded characters are in English. In PDF, definitions are made on an object basis to define the layout of the document. Lines 1 and 38 are a header and a footer indicating the PDF format. Lines 4-11 define page objects. Lines 12-24 define the typeface object being used. The start code of the character defined by line 16 and the end code of the character defined by line 17, and lines 18 to 19 define the character width of the end character from the start. Line 21 is a typeface. Lines 25-37 define the text object to be printed. Line 28 indicates the start of the text object, and line 34 indicates the end of the text object. Line 30 defines Tc as the character spacing and Tw as the word spacing. Line 31 defines the characters to be printed and, if necessary, the placement offset. Line 32 defines an offset to the next line where Td. Line 33 defines the character to be printed.

  In order to extract the character information 126 from the document file 15, first, the font, font, and character width parameters for each character that are common to the text block are extracted from the text block (550). The parameter of the starting point coordinates of the first character is extracted (551). The end point coordinates are calculated from the parameters of the character width and the character interval for each character, and the obtained end point coordinates are set as the start point coordinates of the next character (552). The process of 552 is repeated until the start point coordinates of characters having different coordinates in the line direction are found (553). The character processed in 552 is registered in the character information 126 as a character string 213 on the same line (555). The processes from 551 to 555 are repeated until the text block is completed (556). The processes from 550 to 556 are repeated until the page object is completed (557). The process in the page object is repeated until the document is finished (558).

  In the embodiment of the present invention, the column number 233 is the number of characters from the first character of the line, but it is also possible to use, for example, the start point coordinates in units of characters obtained in FIG. In the embodiment of the present invention, the setting of the link destination URL 223 is set for each line or word, and the line link information 124 and the word link information 125 are respectively created. However, the link destination URL 223 can be set for each sentence. is there.

  Hereinafter, the creation procedure of the sentence link information 590 in which the link destination URL 223 is set for each sentence will be described with reference to the sentence information 580 and the creation procedure of the sentence information 580 with reference to FIGS. 29, 30, and 31. FIG. 29 is a flowchart showing a procedure for creating the sentence information 580 of this embodiment.

FIG. 30 is an example of the sentence information 580 of the present embodiment. The sentence information 580 is composed of sentence data 582-1 to k, and the sentence data 582-1 to k includes a sentence 581 and a sentence 581. The page number 210 of the page and the line number 211 in the character data 212 in which the first character of the sentence 581 exists is set as the starting line number 273, and the column number 233 including the number of characters counted from the left of the character string 213 is set as the starting column. The number is 274. Further, the line number 211 in the character data 212 in which the last character of the sentence 581 exists is set as the end line number 271, and the column number 233 including the number of characters counted from the left of the character string 213 is set as the end line number 272.
In this embodiment, the column numbers 272 and 274 are the number of characters from the left in the character string. However, the coordinates of the start point of the character calculated in FIG. 28 can also be used.

  FIG. 31 is an example of the sentence link information 590 of the present embodiment. The sentence link information 590 is composed of sentence link data 591-1 to 59k, and the sentence link data 591-1 to 59k is sentence data 582-1. Corresponding to ~ k, instead of the sentence 581 of the sentence information 580, the link destination URL 223 associated with the sentence 581 is set.

  The sentence information 580 is created using the character information 126 created in FIG. 28 and the font, font, character code, and character size. For the character string 213 for each row registered in the character data 212-1 to 212-b of the character information 126, the character typeface, character type, character size, etc. are extracted in the same manner as the typeface and character type extracted from the document file 15 in FIG. Then, it is determined whether the font, font, character size, etc. of the next line are different (570). If they are different, the character string 213 is additionally registered in the sentence data 582 of the sentence information 580 as a heading line (573). Next, it is determined whether the first character is clearly different from the preceding and following lines such as the symbol “(1)” and the heading number “1.1” (571). If they are different, the character string 213 is additionally registered in the sentence data 582 of the sentence information 580 as a heading line (573). Next, if it is not a heading line, a phrase “.” Is searched from the character string 213 in the sequential line and registered in the sentence data 582 of the sentence information 580 as a sentence 581 separated by the phrase (572). The above processing is repeated until all lines in the document are scanned (574).

Next, the configuration of the document registration apparatus will be described with reference to FIG. FIG. 4 is an example of a hardware configuration diagram of the document registration apparatus according to the present embodiment.
The correspondence between the blocks in FIGS. 1 and 4 will be described. The input device 45 corresponds to the keyboard 700 and the mouse 701. The display device 40 corresponds to the display 703. The communication device 320 corresponds to the communication circuit 705. The storage device 44 corresponds to the data memory 801. The document selection device 21 and the character recognition device 24 correspond to the document registration program 810.

  The keyboard 700 is used for text input from a registrant. The mouse 701 performs a button operation on the display screen from the user. The external storage device 702 inputs the document file 15 from a storage medium such as a floppy (registered trademark) disk or a compact disk in which the document file 15 is stored, and outputs the data stored in the data memory 801 to the storage medium. . The display 703 displays an operation screen and the like to the registrant. The CPU 704 executes the program in the program memory 800. The communication circuit 705 transfers data stored in the data memory 801 between the server apparatuses 320 via the network 36. The program memory 800 stores a document registration program 810 that executes the operation of each unit in FIG. The data memory 801 includes document information 120, document attribute information 121, word collation dictionary 122, word information 123, line link information 124, word link information 125, character information 126, and temporary data for work used by the document registration program 810. Is stored.

The procedure for the registrant to operate the link destination URL 223 in the document registration apparatus 300 for link destination registration according to this embodiment will be described below with reference to FIG. FIG. 6 is an example of a screen displayed on the display 703 of this embodiment.
The document registration program 810 executed by the CPU 704 displays a screen operated by the registrant on the display 703 and operates the keyboard 700 and the mouse 701 to accept an input operation. The button can be selected using the mouse 701. In the input field, characters can be input by operating the keyboard 700.

  When the document registration program 810 is activated, a document registration screen 600 is displayed first. First, a procedure for registering a document will be described. The document registration screen 600 displays a list of currently registered document information 120. When the registrant wants to add a new document, the registrant can display the document information screen 610 by selecting the add button 601. When deleting the registered registration in the document, by selecting the delete button 603 of the item to be deleted, the selected document information is deleted from the document information 120 and deleted from the list display. When setting the link information of a document, the link registration screen 620 is displayed by selecting the setting button 602 for the item of the document to be set.

  The document information screen 610 displays the automatically set document number 200 in the document number field 611 and inputs the title 201 in the title field 612. The registrant inputs the document file 15 by selecting the open button 613. Document information such as the number of pages of the input document file 15 is displayed in a document information display field 614. The registrant confirms that there is no problem in the registered contents, and selects the registration button 615 to start document analysis and display the document registration screen 600. When canceling registration, selecting the cancel button 616 causes the document registration screen 600 to be displayed without being registered in the document information 120.

Next, a procedure for setting link information will be described. On the link registration screen 620, the registrant enters the link destination URL 223 to be set for the entire document in the link destination field 621, and confirms with the confirm button 622. The registrant displays the link confirmation screen 660 by selecting the confirmation button 623 in the same manner as necessary. The registrant selects the confirmation button 622 to register the document number 223 and the input link destination URL 223 in the document link table 240. When setting link information in units of pages, the page setting screen 630 is displayed by selecting the page batch setting button 624. In setting link information in units of lines, a line setting screen 640 is displayed by selecting a line batch setting button 625. In the link information setting for each word, the word setting screen 650 is displayed by selecting the word setting 626 button. Similarly, when the return button 627 is selected, the previous screen is displayed.
The link confirmation screen 660 displays an image corresponding to the set link destination URL 223 on the display screen 662. If the link destination URL 223 is incorrect, the corrected link destination URL 223 is entered in the link destination field 661 and reflected in the link destination field 621 by selecting the update button 663.

  In the page setting screen 630, the registrant adds a setting item by selecting the add button 601, and inputs the page number 210 to be set in the page number input field 631 of the added setting item and the link destination URL 223 in the link destination field 621. . By selecting the confirmation button 622, the link destination URL 223 and the page number 210 are registered in the page link table 250.

  In the row setting screen 640, the registrant inputs a page number 210 to be set in the page setting field 641, and a setting item is added by selecting an add button 601. The registrant inputs the line number 211 desired to be set in the line number input field 642 of the added setting item and the link destination URL 223 in the link destination field 621. The registrant registers the page number 210, the line number 211, and the link destination URL 223 in the line link table 260 by selecting the confirmation button 622.

  On the word setting screen 650, the registrant inputs a page to be set in the page setting field 641 and a line to be set in the line setting field 651, and a setting item is added by selecting an add button 601. The word contained in the page and line input in the page setting field 641 and the line setting field 651 is acquired from the word information 123 in the word field 652 of the added setting item and displayed in a list. The registrant enters the link destination URL 223 in the link destination field 621 corresponding to the word to be set. The registrant selects the value set in each of the page setting field 641, the row setting field 651, the word field 652, and the link destination field 621 and the column number 233 of the set word from the word information 123 by using the confirm button 622. Register in the link information 125.

  In the embodiment of FIG. 6, an example in which the setting is performed by switching the screen is shown. However, as shown in FIG. 7, it is of course possible to perform all the settings in one screen without changing the screen.

  The operation of FIG. 7 will be described. FIG. 7 is an example of another screen displayed on the display 703 for link destination registration of this embodiment. In FIG. 7, the registration screen 180 displays a document to be set in the document display area 181, and sets a link destination URL 223 for each line in the line link setting field 185 corresponding to the line number 182. In addition, for a word for which the link destination URL 223 can be set, a setting frame 183 is displayed in the document displayed in the document display area 181, and the link destination URL 223 can be set for each word in the word link setting field 185.

  Next, a procedure for creating the row link information 124 of FIG. 11 in the link destination registration unit 111 will be described in detail with reference to the flowchart of FIG. 14 and FIG. FIG. 13 is an example of data created by the link destination registration unit 111 of this embodiment, and FIG. 14 is a flowchart showing the operation of the link destination registration unit 111 of this embodiment.

  In FIG. 13, the document link table 240 is composed of document link data 243-1 to 24-f set for each document, and the document link data 243-1 to 243-f is composed of a document number 200 and a link destination URL 223. The page link table 250 includes page link data 253-1 to 253-g set for each page, and the page link data 253-1 to 253-1 to 25g includes a page number 210 and a link destination URL 223. The line link table 260 includes line link data 263-1 to 263-h set for each line, and the line link data 263-1 to 263-h includes a page number 210, a line number 211, and a link destination URL 223.

  The line link information 124 is automatically generated from the document link table 240, the page link table 250, and the line link table 260. That is, it is inspected for each line whether or not the link destination URL 223 is set in the line link table 260 (280), and when the link destination URL 223 of the inspected line is set, the line link information 124 is inspected. The link destination URL 223 of the line thus set is set (281). If the link destination URL 223 is not set in the inspected line of the line link table 260, it is inspected in the page link table 250 whether the page including the line to be inspected has the link destination URL 223 set (282). When the link destination URL 223 is set, the link destination URL 223 of the designated page including the row to be checked in the page link table 250 is set as the link destination URL 223 of the row to be checked in the row link information 124 (283). When the link destination URL 223 is not set in the designated page of the page link table 250, the link destination URL 223 corresponding to the corresponding document number 200 in the document link table 240 is set as the link destination URL 223 of the line to be checked in the line link information 124. Set (284).

  Details of the mobile terminal device 300 and the server device 320 of FIG. 1 will be described in detail with reference to FIG. Hereinafter, the operation of each unit of the mobile terminal device 300 and the server device 320 in FIG. 15 will be described with reference to the flowchart in FIG. 16 and FIGS. 20 and 21. FIG. 15 is an example of a block diagram illustrating operations of the mobile terminal device 300 and the server device 320 according to the present embodiment, and FIG. 16 is a flowchart diagram illustrating operations of the mobile terminal device 300 and the server device 320 according to the present embodiment. is there. FIG. 20 shows an example of a printed document according to the present embodiment, and FIG. 21 shows an example where an image photographed by the camera 303 according to the present embodiment is displayed on the display 716.

The correspondence between the blocks in FIGS. 1 and 15 will be described. The photographing device 23 corresponds to the camera 303. The display device 22 corresponds to the display unit 305. The document selection device 21 corresponds to the document selection unit 301. The character recognition device 24 corresponds to the character recognition unit 304, the word collation unit 322, and the recognition setting unit 302.
In FIG. 15, the server device 320 first obtains document information 120, document attribute information 121, word collation dictionary 122, line link information 124, word link information 125, and character information 126 from the document registration device 10 (355). This process only needs to be executed once at the time of document registration.

  In the mobile terminal device 300, the document selection unit 301 copies information selected by the user from the document information 120 of the server device 320 to the document selection information 127 (355). Further, the user selects a search target document from the document selection information 127 (340). The document management unit 321 of the server device 320 selects the document attribute information 121, the word matching dictionary 122, the line link information 124, the word link information 125, and the character information 126 of the document selected by the user (350). The recognition setting unit 302 sets parameters of the character recognition unit 304 from the document attribute information 121 selected by the document management unit 321 and copies the word matching dictionary 122 to the word matching dictionary 307 (341). The camera 303 images a partial area 371 of the document 370 printed by the user's operation. The character recognizing unit 304 recognizes characters included in the image captured by the camera 303, and performs the character in the center line 385 including the marker 841, the line 386 above the center line 385, and the line 387 below the center line 385. And character candidates are output (343). The word collation unit 322 uses the words registered in the word collation dictionary 122 from the character candidates in the center line 385, the upper line 386, and the lower line 387 to extract the attention word 382, the upper word 383, and the lower word 384. Output (352).

  In the server device 320, the search unit 323 searches for the position of the word from the character information 126 using the attention word 382, the upper word 383, and the lower word 384, and registers the certainty of search in the candidate list 128 (353). . The determination unit 324 searches the link destination URL 223 using the line link information 124 and the word link information 125 from the word and the word position with the highest certainty factor from the candidate list (354). The display unit 305 of the mobile terminal device 300 displays the information of the link destination URL 223 searched by the determination unit 324 using a browser (344).

  Next, details of the word position search of the search unit 323 will be described below with reference to the flowchart of FIG. 23 and FIG. FIG. 23 is a flowchart showing the operation of the search unit 323 of this embodiment, and FIG. 24 is an example of the candidate list 128 of this embodiment.

  In FIG. 24, the candidate list 128 includes candidate data 425-1 to i, and the candidate data 425-1 to i includes a page number 210, a row number 211, and a column number 233 indicating the position of the word of interest 382 in the document. And a certainty factor 424 indicating the possibility of matching with the location of the attention word 382.

  The search unit 323 searches the word information 123 for the attention word 382, and acquires the row number 211 and the column number 233 of the same word 230 as the attention word 382 (400). The same word 230 as the upper word 383 and the lower word 384 is searched from words included in the upper and lower lines of the word 230 searched by the word information 123 (401), and the existence of the upper word 383 and the lower word 384 is determined. (402). If the upper word 383 and the lower word 384 are not found, the search is continued from the next word in the word information 123. When the upper word 383 or the lower word 384 exists in the word information 123, the positional relationship between the attention word and the upper word 383 and the lower word 384 is determined (403). If the positional relationship is within the threshold, the character position of the attention word is calculated and registered in the candidate list 128 together with the certainty factor 424 (404). The certainty factor 424 of the candidate list 128 is determined by the difference of the column number 233 between the attention word 382 and the upper word 383 or the lower word 384 and the presence of the upper word 383 or the lower word 384. The search for the character position of the word of interest 382 is repeated for all the word information 123 (405).

  In the embodiment of the server device 300 of the present invention, the search unit 323 searches for the character string position using the word information 123, but the character information 126 can also be used. That is, instead of obtaining the column number from the word information 123, the column number can be obtained by searching for a word from the character string of the character information 126 and sequentially calculating the word position from the number of characters in the character string.

  Next, details of the operation of the determination unit 324 will be described below with reference to the flowcharts of FIGS. 11, 12, 21, 22, 24, and 25. FIG. 22 is another example in which an image photographed by the camera 303 of the present embodiment is displayed on the display 306, and FIG. 25 is a flowchart showing the operation of the determination unit 324 of the present embodiment.

  The determination unit 324 searches candidate data 425-n having the highest certainty factor 424 from the candidate data 425-1 to i in the candidate list 128 (365). It is determined whether the marker 381 is set on the attention word 382 (361). When there is a marker 381 on the attention word 382, the word link that matches the page number 210, the row number 211, and the column number 233 of the candidate data 425-n from the word link data 235-1 to 235-1e of the word link information 125. Data 235-m is searched (362). Next, the attention word 382 and the word 230 of the word link data 235-m are collated (363). If they match, the link destination URL 223 of the word link data 235-m is output (364). If they do not match, and the determination of the attention word 382 (361), as shown in FIG. 22, the marker 381 and the attention word 382 are misaligned, the candidates are obtained from the line link data 222-1 to 22b of the line link information 124. The line link data 222-r in which the page number 210 and the line number 211 of the data 425-n match is searched (365). The searched link destination URL 223 is output (364).

  In the embodiment of the present invention, the determination unit searches the link destination URL 223 in units of words or units using the word link information 125 and the line link information 124, but the link destination URL 223 in units of sentences from the sentence link information 590. It is also possible to search. In the search for the link destination URL 223 for each sentence, first, the page number 210, the line number 211, and the column number 233 indicating the position of the word of interest 382 in the document are acquired from the word information 123. Next, it is the same as the acquired page number 210, the line number 211 is within the line number 273 of the start point of the character and the line number 271 of the end point, and the column number 233 is the column number 274 and the end point of the start point of the character. The link destination URL 223 in the sentence link information 590 within the column number 272 is output. The link destination URL is used as the link information of the sentence. For example, a Japanese sentence can be output as an English translation sentence, and an English sentence can be output as a Japanese translation.

  A hardware configuration of the mobile terminal device 300 will be described with reference to FIG. FIG. 17 is an example of a hardware configuration diagram of the mobile terminal device 300 of the present embodiment. The correspondence between the blocks in FIGS. 1 and 17 will be described. The input device 20 corresponds to the input key 710. The photographing device 23 corresponds to the camera 303. The display device 22 corresponds to the display 716. The communication device 25 corresponds to the communication circuit 713. The character selection device 21 and the character recognition device 24 correspond to the character recognition program 820.

  In FIG. 17, an input key 710 performs selection of a document to be searched and camera setting. The communication circuit 713 is connected to the communication circuit 714 of the server device 320 via the network 36, and transmits a character string obtained by character recognition from an image captured by the camera 303 and receives information related to the character string. Do. A display 716 displays an image captured by the camera and selects a document. The character recognition program 820 is a program for operating the mobile terminal device 300. The data memory 801 stores document selection information 127 and a word matching dictionary 307.

  The hardware configuration of the server device 320 will be described with reference to FIG. FIG. 18 is an example of a hardware configuration diagram of the server apparatus 320 according to the present embodiment. The correspondence between the blocks in FIGS. 1 and 18 will be described. The communication device 30 corresponds to the communication circuit 714. The communication circuit 33 corresponds to the second communication circuit 715. The position search device 34 and the information search device 31 correspond to the document search program 840. The storage device 32 corresponds to the data memory 801.

  In FIG. 18, the second communication circuit 715 is connected to the document registration apparatus 10 via the network 26, receives data stored in the data memory 801 of the document registration apparatus 10, and stores it in the data memory 801 of the server apparatus 320. To do. The document search program 840 is a program that operates the server device 320. The data memory 801 stores the received document information 120, document attribute information 121, word matching dictionary 122, line link information 124, word link information 125, word information 123, and candidate list 128.

  The transition of the screen displayed on the display 716 of the mobile terminal device 300 will be described in detail below with reference to FIG. FIG. 19 is an example of a screen transition diagram displayed on the display 306 of this embodiment. The following buttons are selected by the input key 710. The previous screen is displayed by selecting the return button 913. When activated, the character recognition program 820 displays a document selection screen 900 on the display 716. The document selection screen 900 reads out available documents from the document selection information 127 and displays them in the document list 901. When a document to be used is not displayed in the document list 901, the user displays an add document screen 910 by selecting an add button 902. If unnecessary document information is to be deleted from the document selection information 127, the corresponding document information is deleted from the document selection information 127 by selecting a document to be deleted from the document list 901 and selecting a delete button 903. When a search is to be executed, a document confirmation screen 930 is displayed by selecting a document from the document list 901 and selecting a selection button 904. The document addition screen 910 displays a list of documents registered in the document information 120 in the document list 911, selects a document to be added to the document selection information 127 from the document list 911, and selects a detail button 912 to select a document. A detail screen 920 is displayed. The document details screen 920 displays detailed information of the selected document on the details display 921.

  The user adds the document information of the selected document to the document selection information 127 by confirming the title and billing information and selecting the registration button 922. The document confirmation screen 930 displays the detailed information of the selected document on the detail display 931. The user confirms the title, billing information, etc., and if there is no problem, selects the confirmation button 932 to display the shooting screen 940. indicate. The shooting screen 940 displays an image to be shot on the display image 380, and the user performs a search by selecting a search button 943 according to a character to search for the marker 381, and displays a confirmation screen 950 when the search is completed. To do. The confirmation screen 950 displays the searched link destination URL 223 in the link information display 951. If there is no error in the search result, the user selects the display button 952 and displays the browser 960. The browser 960 displays information on the link destination URL 223.

  In the portable terminal device 300 and the server device 320, another method of word matching will be described in detail with reference to the block diagram of FIG. 26 and the flowchart of FIG. 26 is a block diagram illustrating other operations of the mobile terminal device 300 and the server device 320 according to the present embodiment. FIG. 27 is a flowchart illustrating other operations of the mobile terminal device 300 and the server device 320 according to the present embodiment. is there.

  The mobile terminal device 300 includes a first word matching dictionary 501 in advance. The second word matching dictionary 506 is created by the difference between the word matching dictionary 122 and the first word matching dictionary 501. By selecting a document (340), a second word matching dictionary 501 corresponding to the selected document is loaded into the data memory 801 (521). The image input from the camera 303 is recognized by the character recognition unit 304 and a character candidate is output (343). The first word matching unit 500 uses the first word matching dictionary 501 to match words from character candidates (514), and whether the word registered in the first word matching dictionary 501 matches the character candidate. Determine (515). When it matches with the word of the 1st word collation dictionary 501, a word is transmitted to the search part 323 of the server apparatus 320, and the search part 323 searches the position in the document of a word (354). If they do not match, the second word matching unit 505 of the server device 320 matches the words using the second word matching dictionary 506 (522). In the embodiment of the present invention, an example in which the second word matching dictionary 506 is included in the server device 320 has been shown. However, the second word matching dictionary 506 is downloaded to the portable terminal device 300 and the first word matching dictionary 501 and the second word matching dictionary 501 are downloaded on the portable terminal 300 side. It is also possible to perform word matching in the word matching unit 322 using the word matching dictionary 506.

The act of the details of the character recognition section 304 and recognition setting unit 302 will be described below with reference to FIG. 3 3 and 3 4. Figure 3 3 is an example diagram showing an operation of the character recognition portion 304 and the recognition setting unit 302 of the present embodiment, FIG. 3. 4 is an example of the document attribute information 121 of the present embodiment. The document attribute information 121 includes a plurality of document attributes 73-1 to 73-j. The document attributes 73-1 to 73-j are represented by a document number 200, a binarization mode 70, a character direction 71, and a character code 72. Constitute.

  Based on the value of the binarization mode 70 corresponding to the document number 200 of the selected document from the document attribute information 121 received from the server device 320, the recognition setting unit 302 uses the first image binary of the binarization processing unit 51. Switching between the digitizing process 60 and the second image binarizing process 61. Further, the vertical writing processing unit 62 and the horizontal writing processing unit 63 of the character line cutout unit 52 are switched according to the value of the character direction 71. Further, according to the value of the character code 72, the identification unit 64 is set to be compared with the feature amount of the character code designated by the character code 72 in the identification dictionary 65. In the identification processing unit 55, the identification unit 64 compares the feature amount for each character of the feature extraction unit 54 with the feature amounts of all the character codes registered in the identification dictionary 65. The determination unit 66 outputs the character codes in ascending order of comparison difference.

  The usage mode of the document information retrieval system of the present invention will be described in detail below with reference to FIG. FIG. 35 is an example of a system usage mode for operating the document information registration system of this embodiment. The publisher 956 receives a manuscript from the author of the document to be published, inputs it into the DTP 951 from the keyboard 950, and edits it. The DTP 951 outputs to the PDF file 952 for proofreading of the document, and the proofreading document 954 is output by the printer 953. After the proofreading is completed, the publishing print data 955 is output from the DTP 951, and the sales book 957 is printed by the printing machine 956. The document registration apparatus 10 receives a PDF file 952 output for proofreading, and creates a search DB in which related information is added to the document. The created DB is transferred to the server apparatus 300 of the content provider 958 via the network 26. The user purchases the published book 957 and takes a partial image of the book 957 using the mobile terminal device 320. The mobile terminal device 320 transfers the character string recognized from the image to the server device 300 via the network 36. The server device 300 searches for information related to the character string from the search DB. The retrieved information is displayed on the mobile terminal device 300.

  The publisher 960 inputs a book 957 that has already been published as an image from the scanner 961. The OCR 962 recognizes characters from the image of each page and also detects the position of the character image. The structured 958 creates a PDF file 952 from characters and character positions. The document registration apparatus 10 inputs the PDF file 952 and creates a search DB in which related information is added to the document.

The procedure for creating the search DB from the PDA file described above may of course be undertaken by a third party other than the publisher.
In the embodiment of the present invention, a URL is assumed as information related to a document. However, it is of course possible to directly send information such as characters, sounds, and images in addition to the URL.

  It is intended for Japanese documents, but of course it is easy to adapt to other languages such as English. For example, in English and Korean, words are separated by spaces, so they can be easily extracted in units of words without performing morphological analysis. It is also possible to search the position of the attention word using the word.

  In the embodiment, the mobile terminal device and the server device are divided, but the sharing of processing is not limited. For example, although word collation was implemented with the portable terminal device, it is also possible to implement with the server device. For example, although character recognition was implemented with the portable terminal device, it is also possible to implement with the server device. Moreover, although it divided into the server apparatus and the portable terminal device, the operation | movement of a server apparatus and the operation | movement of a portable terminal can also be performed within the same apparatus.

  In the embodiment, the character recognition program 820 is stored in advance in the program memory 800 of the mobile terminal device 320. However, the character recognition program 820 can be downloaded from a server and executed. In the embodiment, the search target document is selected. However, the character recognition program 820 corresponding to the document can be downloaded in accordance with the search target document.

In the embodiment, three words of the attention word 382, the upper word 383, and the lower word 383 are used for the word position search, but the number of words used for the search is not limited.
In the embodiment, it is also possible to acquire characters and character positions using, for example, an OCR for a document that has been input from an electronic file but has already been printed.

  In the embodiment, the link destination URL 223 is set in units of lines or words, but the link destination URL can be set in units of sentences. For example, as shown in FIG. 28, the searched word is set by memorizing the link destination URL 223 by manipulating the line number 211 and the column number 233 at the start of the sentence and the line number 271 and the column number 272 at the end of the sentence. It is also possible to output the corresponding link destination URL 223 if it is within the start and end of the sentence. As a result, for example, it is of course possible to use English translations for Japanese sentences and Japanese translations for English documents.

The example of the whole block diagram of this embodiment. FIG. 3 is an exemplary block diagram illustrating an operation of the document registration apparatus according to the embodiment. The flowchart figure which shows operation | movement of the document registration apparatus of this embodiment. 2 is an example of a hardware configuration diagram of a document registration apparatus according to the present embodiment. An example of a document file input in the present embodiment. An example of a screen displayed on the display 703 of the present embodiment. The example of the other screen displayed on the display 703 of this embodiment. The example of the document information 120 of this embodiment. The example of the character information 126 of this embodiment. The example of the word information 123 of this embodiment. The example of the line link information 124 of this embodiment. The example of the word link information 125 of this embodiment. The example of the data which the link destination registration part 111 of this embodiment produces. The flowchart figure which shows operation | movement of the link destination registration part 111 of this embodiment. The example of the block diagram which shows operation | movement of the portable terminal device 300 and the server apparatus 320 of this embodiment. The flowchart figure which shows operation | movement of the portable terminal device 300 and the server apparatus 320 of this embodiment. The example of the hardware block diagram of the portable terminal device 300 of this embodiment. The example of the hardware block diagram of the server apparatus 320 of this embodiment. The example of the screen transition diagram displayed on the display 716 of this embodiment. An example of a printed document according to the present embodiment. An example in which an image photographed by the camera 303 of the present embodiment is displayed on the display 716. Another example in which an image photographed by the camera 303 of the present embodiment is displayed on the display 716. The flowchart figure which shows operation | movement of the search part 323 of this embodiment. The example of the candidate list | wrist 128 of this embodiment. The flowchart figure which shows operation | movement of the determination part 324 of this embodiment. The block diagram which shows other operation | movement of the portable terminal device 300 of this embodiment, and the server apparatus 320. FIG. The flowchart figure which shows the other operation | movement of the portable terminal device 300 and the server apparatus 320 of this embodiment. The flowchart figure which shows the preparation procedure of the character information 126 of this embodiment. The flowchart figure which shows the preparation procedure of the sentence information 580 of this embodiment. The example of the sentence information 580 of this embodiment. The example of the sentence link information 590 of this embodiment. The example of the character recognition process of this embodiment. The example of the figure which shows operation | movement of the character recognition part 304 and the recognition setting part 302 of this embodiment. An example of the document attribute information 121 of the present embodiment. An example of a system usage mode for operating the document information registration system of the present embodiment.

Explanation of symbols

10 document registration device, 20 input device, 21 document selection device, 22 display device, 23 photographing device, 24 character recognition device, 25 communication device, 26 network,
30 communication device, 31 information search device, 32 storage device, 33 communication device, 34 position search device, 36 network,
41 document input device, 42 analysis device, 43 registration device, 44 storage device, 45 input device, 46 communication device,
300 server device, 320 mobile terminal device.

Claims (11)

An input means for inputting a document file describing a document to be searched;
A character string extracting means for extracting a character string in the document of the document file;
And location registration means for registering the character position location in said document of the string,
Information registration means for registering related information indicating information linked to the character string;
Storage means for storing in association with the related information and the character position location with the character string,
The target character string specified from an image that is a part of the document, and the target character string input from a terminal device that includes a character recognition unit that recognizes a peripheral character string around the target character string and the peripheral based on the positional relationship between the character strings, and character position search means for searching said character position in said document of the target character string satisfying the positional relationship from the storage means,
Information retrieval means for retrieving the related information linked to the character string at the retrieved character position;
A document information search system comprising: output means for outputting the related information searched by the information search means to the terminal device.   The document information search system according to claim 1, wherein the character string extraction unit extracts a parameter of a start point coordinate of a first character, calculates an end point coordinate from parameters of a character width and a character interval for each character, A document information retrieval system for extracting the character string.   2. The document information retrieval system according to claim 1, wherein the character string extraction means includes morpheme analysis means for dividing the character string into morphemes, and extracts the morphemes as character strings. .   4. The document information retrieval system according to claim 3, wherein the morpheme analyzing unit extracts a character string obtained by selecting and removing unnecessary morphemes from the morpheme.   4. The document information search system according to claim 3, wherein the character string extraction unit extracts a character string obtained by dividing a character string constituting the morpheme or synthesizing character strings constituting adjacent morphemes.   2. The document information search system according to claim 1, wherein the character string extracting means extracts parameters of a typeface, a font and a character size, and extracts a headline from the parameters, and determines a punctuation point in the character string. A document information retrieval system comprising sentence extraction means for extracting a sentence and extracting the headline and the sentence as the character string.   2. The document information search system according to claim 1, wherein the document information search system further includes document attribute information extraction means for extracting character attribute and character direction document attribute information of the document, and a document attribute for storing the document attribute information. A document information search system comprising storage means, wherein the terminal device refers to the document attribute information in character line segmentation for character recognition in the character recognition means in the character recognition means.   2. The document information retrieval system according to claim 1, wherein the document information retrieval system further includes a word collation dictionary that stores words to be collated, the word collation dictionary for character recognition at the terminal device. A document information retrieval system characterized by being used for character line segmentation. The document information retrieval system according to claim 1, wherein the terminal device further includes a first word matching dictionary created with a specific character string,
The document information search system includes a second word matching dictionary that stores a character string that is a difference between the character string extracted from a document to be registered and a character string registered in the first word matching dictionary. ,
A document information retrieval system that outputs a character string using the second word matching dictionary when a character string registered in the first word matching dictionary is not retrieved.   10. The document information search system according to claim 1, wherein the portable terminal includes a document selection unit for selecting a document to be searched.   2. The document information search system according to claim 1, wherein the information search means searches for the linked related information in units of words or lines of the character string at the character position.

Images