JP2006053622A - Document link information acquisition system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide link information associated with a document from an image including a part of the document acquired by a portable information terminal. <P>SOLUTION: This document link information acquisition system is provided with a storage device holding a sentence consisting of words and the link information concerned with the sentence in association with each other and storing a translation dictionary holding associated arrangement information of the respective words included in the sentence, a character recognition part performing character recognition processing on a target word pattern designated by a user and on a word pattern around the target word pattern from an image taken by an image photographing part for outputting a character code of the word pattern, an extraction part extracting the relative arrangement information about the target word pattern designated by the user and the word pattern around the target word pattern, and a search part searching a database holding the associated link information about the sentence by using the character code obtained by the character recognition part and the relative arrangement information obtained by the extraction part about the target word pattern and the word pattern around the target word pattern. A result acquired by referring to the link information obtained by the search part is displayed on a display part. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a document link information system, and more particularly to a document link information system that obtains related link information by recognizing characters included in an image captured by a portable information terminal and a related link information acquisition processing method.

  Various techniques for recognizing characters in a character string included in an image taken by a portable terminal such as a mobile phone equipped with a camera and translating the character text as a recognition result have been proposed. For example, Patent Document 1 (Japanese Patent Application Laid-Open No. 09-138802) has a character recognition function and a translation function inside a portable terminal, and uses these functions to recognize a character string in an image taken by a camera. A translation system for translation processing is disclosed.

  In addition, as a recognition technology application service for images taken with a mobile terminal such as a mobile phone equipped with a camera, a two-dimensional code printed on paper is read with a camera and link information such as a URL included in the code is acquired. It is known that a linked WEB site can be accessed and data can be acquired from the site. Especially in Japanese camera-equipped mobile phones, 2D code recognition software is installed, and 2D codes printed on paper such as magazines and advertisements are read by the phone's camera and the URL stored in the code etc. In general, a service for acquiring link information and connecting to a WEB server is performed.

  Alternatively, the OCR (Optical Character Reader) software is installed in the camera-equipped mobile phone, the URL character string image is taken by the camera, and then the function of recognizing the character and accessing the WEB server is installed in the phone. ing.

JP 09-138802 A

  By printing the two-dimensional code on the paper in this way, it becomes possible to link the information on the paper and the information on the Internet called WEB via the information terminal with camera. However, if you try to print a 2D code on paper, you may not be able to secure a printing space on the paper, or the printing will tend to be avoided if the layout of the paper is damaged due to the unique pattern and size of the 2D code. there were.

  When a URL character string is printed on a page, it is possible to refer to a related WEB site on the page by manually entering the URL. However, it is very burdensome for the user to input the URL character string. . On the other hand, it is possible to reduce a user's burden of inputting a URL by photographing the URL with a camera and recognizing the character of the image. However, due to camera resolution, it is necessary to shoot in the close-up mode, which is greatly influenced by camera shake when recognizing the URL, and it is difficult to stably match the camera to the URL character string, and still a great burden on the user. I was forced to.

  In addition, it is necessary to correct a misread character when a character string recognized by the user of the telephone is visually confirmed after photographing and then found. If a URL is accessed without correcting misread characters, the site cannot be correctly accessed, so it is necessary to check carefully, and this work also increases the burden on the user.

  An object of the present invention is to shoot a part of a document with a camera using a portable information terminal, identify a location in the document where a character string included in the captured image exists, and associate the location with the location. An object of the present invention is to provide a document link information acquisition system and a link information acquisition method capable of acquiring link information such as a URL. That is, a system capable of adding link information to an arbitrary place where characters are printed on a paper without printing a special character code such as a two-dimensional code or a character string representing link information such as a URL on the paper. It is to provide a method.

The present invention relates to a character included in an image photographed by the image photographing unit using a portable information terminal having an image photographing unit, an input unit that is operated by a user to input information, and a display unit. In a document link information acquisition system that obtains link information and refers to the link information and displays link destination data on the display unit, a sentence composed of words and link information related to the sentence are stored in association with each other. In addition, a storage device that stores a document link information database that associates and holds arrangement information of each word included in the sentence, a focused word pattern specified by the user from an image captured by the image capturing unit, and the focused A character recognition unit that performs character recognition processing on a word pattern around the word pattern and outputs a character code of the word pattern, and a target word pattern specified by the user And an extraction unit that extracts relative arrangement information about the word pattern around the target word pattern and the word recognition unit obtained about the word pattern around the target word pattern and the target word pattern. A search unit that searches the document link information database using the character code and the relative arrangement information obtained from the extraction unit, and links with reference to the link information obtained from the search unit The document link information acquisition system is characterized in that the previous data is displayed on the display unit.
Preferably, the document link information database unit stores an English sentence and its link data in association with each other.

  In another preferred example, the document link information database unit stores a sentence composed of words separated by spaces and link information related to the sentence in association with each other.

  According to an example, the relative arrangement information is an angle of a line segment drawn from the rectangular centroid of the focused word pattern specified by the user to the rectangular centroid of each peripheral word pattern of the focused word pattern.

  In another example, the relative arrangement information is relative arrangement information of a line including a target word pattern specified by a user and a line including each peripheral word pattern of the target word pattern. is there.

In another preferable example, the document link information acquisition system according to the present invention is included in an image capturing unit, an input unit that is operated by a user to input information, and an image captured by the image capturing unit. A portable information terminal having a link information acquisition unit that performs processing for obtaining link information associated with characters, and a display unit that displays link destination data with reference to the link information obtained by the link information acquisition unit In the document link information acquisition system used, the link information related to each sentence in the document is stored in association with each other,
A storage device that stores a document link information database that associates and holds arrangement information of each morpheme included in each sentence, a focused morpheme pattern designated by the user from an image captured by the image capturing unit, and the focused morpheme pattern A character recognition unit that performs character recognition processing on the surrounding morpheme patterns and outputs a character code of each morpheme,
An extraction unit that extracts relative arrangement information regarding a morpheme pattern specified by the user and a morpheme pattern existing around the morpheme pattern of interest, and a morpheme pattern existing around the morpheme pattern of interest and the morpheme pattern of interest A search unit that searches the document link information database using the character code obtained from the character recognition unit and the relative arrangement information obtained from the extraction unit, and from the search unit A document link information acquisition system that displays link destination data on the display unit with reference to the obtained link information.

In still another preferred example, the document link information acquisition system according to the present invention uses an image capturing unit, an input unit that is operated by a user to input information, and a portable information terminal having a display unit. In a document link information acquisition system that obtains link information associated with characters included in an image photographed by the image photographing unit and displays link destination data on the display unit with reference to the link information, the word information A storage device that stores a document link information database that holds the link information related to the sentence and the link information related to the sentence in association with each other and holds the arrangement information of each character included in the sentence, and the image capturing unit Character recognition processing is performed on the target character pattern designated by the user from the image and the character pattern around the target character pattern, and the character pattern A character recognition unit that outputs a character code; an extraction unit that extracts relative arrangement information regarding a target character pattern designated by a user and a character pattern around the target character pattern; and the target character pattern and the target character A search unit that searches the document link information database using the character code obtained from the character recognition unit and the relative arrangement information obtained from the extraction unit with respect to a character pattern around the pattern;
The document link information acquisition system displays the link destination data on the display unit with reference to the link information obtained from the search unit.

  As another example of the document link information acquisition system, the present invention is a document link information acquisition system including a storage device that stores the document link information database and a server that includes the search unit, and the portable information terminal The character code obtained from the character recognition unit and the relative arrangement information obtained from the extraction unit are transmitted to the server via a network, and the document link information is transmitted by the search unit in the server. Link information obtained as a result of searching the database is transmitted to the portable information terminal via the network.

  Furthermore, as another example, there is provided a document link information acquisition system including a server including the character recognition unit, the extraction unit, a storage device that stores the translation dictionary, and the search unit, and an image photographing unit of a portable information terminal The obtained image is transmitted to the server via the network, and the link information obtained as a result of searching the document link information database by the search unit in the server is transmitted to the portable information terminal via the network.

  In a preferred example, the document link information database in the document link information acquisition system stores a word index part for searching a sentence including a word, geometric information of a word included in the sentence, and a character code of the word. And a link information storage unit that stores link information related to each sentence.

The present invention is also understood as a characteristic portable information terminal or server used in the document link information acquisition system.
Furthermore, in the document link information acquisition system, the present invention is grasped as a program executable on a computer for realizing the function of the extraction unit.

Furthermore, the present invention is grasped as a document link information acquisition method. For example, a link associated with characters included in an image photographed by the image photographing unit using a portable information terminal having an image photographing unit, an input unit operated by a user to input information, and a display unit In the document link information acquisition method for obtaining information and referring to the link information and displaying the link destination data on the display unit, the sentence composed of words and the link information related to the sentence are held in association with each other, A step of preparing a document link information database in association with the arrangement information of each character included in the sentence in a storage device, and a target character pattern designated by a user from an image photographed by the image photographing unit; A step of performing character recognition processing on a character pattern around the target character pattern and outputting a character code of the character pattern, and a call specified by the user A step of extracting relative arrangement information regarding a character pattern and a character pattern around the character pattern of interest, and the character obtained by the character recognition unit with respect to the character pattern around the character pattern of interest and the character pattern of interest. A step of searching the document link information database using the code and the relative arrangement information obtained from the extraction unit, and displaying the link destination data with reference to the link information obtained as a result of the search A document link information acquisition method.
In a preferred example, as a result of searching the document link information database, a step of displaying a sentence that is one or more link information candidates including the character pattern of interest on the display unit; Obtaining link destination data from the document link information database by referring to link information corresponding to the candidate designated by the user via the input unit from among the candidates, the document link information The link destination data obtained from the database and the sentence are displayed in correspondence with each other.

According to the present invention, without printing a special character string such as a two-dimensional code or a character string representing link information such as a URL on a paper surface, link information associated with a document on the paper surface is acquired, It is possible to obtain link destination data by referring to the acquired link information. In addition, it is not necessary to photograph the entire sentence to be referred to for link information with a camera, and link information related to the sentence by using words and word arrangement information included in a partial image obtained by photographing a part of the sentence. The link destination data can be obtained by referring to the link information.
As a result, the degree of freedom of the layout of the paper is increased as compared to the case where a character string representing link information such as a special code such as a two-dimensional code or URL or the like is printed on the paper. Furthermore, compared to shooting the entire sentence, the captured image only needs to include any part of the link information reference target sentence, so the degree of freedom of shooting is increased and the usability for the user is greatly increased. improves.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The first embodiment shows a link information acquisition system associated with English sentences. It is possible to electronically hold a document database with the same content as a document printed on paper in advance, and to search the document database using the text of the sentence that you want to refer to and specify the location of the reference sentence . If a link is given to the sentence specified by the search, that is, if the document database is converted into hypertext, the link data corresponding to the searched sentence is referred to It becomes possible to acquire. Therefore, if you try to search the document database using the recognition result text by photographing the entire sentence you want to refer to with the camera, misreading occurs due to various factors such as image distortion, and it is difficult to input the text correctly. It is.

  Therefore, when a user captures an image (partial image) including a part of words included in the reference object, instead of the entire English sentence or any arbitrary plurality of sentences to be referred to, it is included in the partial image. After a word is recognized, the link reference target English sentence stored in the database is searched using the word as a key. Further, the link information associated with the English text of the search result is searched, and the link destination data is displayed on the screen to acquire the link information for each sentence.

  In the present embodiment, in order to search for a reference English sentence stored in the database using the partial image as a key, arrangement information of each character included in the English sentence is also stored in advance in the database. Thereafter, the character code and the arrangement information of the word included in the taken partial image are extracted by the character recognition process. With respect to the character code and arrangement information of the extracted word, the character code and arrangement information of the English word stored in the database are searched. As a result of the search, the English text including the word with the most similar arrangement to the extracted English word arrangement information is retrieved from the database and acquired. After that, the link information stored in association with the searched English sentence is acquired.

The arrangement information used is preferably information that does not depend on the shooting angle and scale as much as possible. For example, it is relative arrangement information and angle information of a line including each word. Alternatively, it is also possible to use character and background color information, character size, ruled line layout, and relative arrangement relationship between ruled lines and characters.
As an example of the word arrangement information, regarding the procedure for extracting angle information representing the relative positional relationship between words included in the partial image, for example, the attention word “as” specified by the user, the up / down / left / right of the attention word, etc. With respect to a plurality of surrounding words (hereinafter referred to as neighboring words), the character code of the word and the rectangular barycentric coordinates of each word are extracted by character recognition processing. Thereafter, line segments 1101 to 1105 are drawn from the rectangular centroid of the word of interest to the rectangular centroids of the surrounding words, and angles 1106 to 1110 of the line segments are calculated. This angle calculation may be performed between any neighboring word centroids. Since the angle between the centers of gravity does not depend on the scale of the image, it can be stable arrangement information. (These will be described in detail later with reference to FIG. 11).
Hereinafter, the first embodiment will be described in detail with reference to FIGS.

  The first embodiment is an example of a system that realizes the above functions and configurations in a portable information terminal.

  FIG. 1 is a block diagram illustrating a configuration example of a portable information terminal.

  The portable information terminal 100 is, for example, a mobile phone, and its main body includes an image photographing unit 110 such as a camera or a scanner that optically inputs an English textbook to be recognized, a recognition target image or character recognition result, a cursor, and the like. The display unit 120 includes a liquid crystal display unit 120 for displaying 172 and the like, and an input unit 130 on which keys 174 to 179 for inputting information by key operation by a user are arranged. Furthermore, as its internal configuration, a terminal control unit 140 that performs overall control of the portable information terminal, a character recognition unit 150 that recognizes words included in an image obtained from the image photographing unit 110, and a character recognition result are used. A link DB search unit 160 for searching for link data corresponding to English sentences, and a memory 180 for storing various processing result data and databases.

  The terminal control unit 140 includes a communication control unit 141, a camera control unit 142 for controlling the image capturing unit 110, a key input control unit 143 for controlling the input unit 130, and a display control unit for controlling the display unit 120. 144, a character recognition / link DB search control unit 145 for controlling the character recognition unit 150 and the link DB search unit 160. The character recognition unit 150 performs word recognition within the rectangular coordinates of the input image based on the output of the word rectangular coordinate extraction unit 151 and the word rectangular coordinate extraction unit 151 for extracting the rectangular coordinates of each word from the input image. A recognition unit 152 is included.

  The memory 180 includes image data 181 captured by the operation of the input unit 130, word recognition result data 182 storing a recognition result by the character recognition unit 150, and link reference candidate sentence search result data storing an output of the link DB search unit 160. 183, a link DB (database) 184 that stores English documents to be photographed and link data corresponding to each English sentence, and link reference data 185 that stores the result of referring to the link data. The link DB 184 also stores arrangement information of each word included in the English sentence.

  Here, the character recognition unit 150 and the link DB search unit 160 may be functions realized by executing software.

  The link DB 184 may be based on the assumption that the ROM in which the link DB is preinstalled is mounted on the portable information terminal 100. Alternatively, it may be assumed that a memory medium such as an SD (Secure Digital) card storing the link DB is purchased and the link DB is installed in the link DB 184 of the memory 180 from the SD card. The link DB 184 is not limited to a static storage device such as a semiconductor memory, and may be stored in a storage device such as a hard disk device.

  FIG. 2 shows a configuration in which a mobile terminal and a WEB server 202 storing link data reference destination data are connected via a network 201. Data obtained by accessing the WEB server 202 is stored in the link reference data 185.

  Next, an operation for accessing and referring to link data associated with each English sentence will be described with reference to a series of processing flows shown in FIG.

  In the description of the overall processing operation according to FIG. 3, the detailed description of each operation may be further referred to FIG.

  First, the user uses the image capturing unit 110 of the portable information terminal 100 to capture an image of a magazine or book that is a character recognition target. The photographed image is stored as a digital image in the area of the image data 181 in the memory 180 (301).

  FIG. 4 shows an example of an English document that is a link data reference target. “P24” and the page number are attached in the upper right. A portion surrounded by 401 indicates a range of an image captured by the image capturing unit 110 and is stored in the image data 181. This image data 181 is a range of an image input to the character recognition unit 150. In this example, it is assumed that an image is captured in order to refer to link information relating to the second sentence starting with “To meet” on the third line and ending with “quality products.” On the fifth line. In order to refer to the link information of this document, in the present embodiment, an arbitrary word constituting the second English sentence can be a target of attention. Here, as shown in the display unit 120 of FIG. 1, it is assumed that the “+” mark 172 indicating the target of interest overlaps “as”.

  Next, the rectangular coordinate indicating the location of the English word is extracted from the input image by the word rectangular coordinate extraction unit 151 (302). Here, as shown in FIG. 5, not only “as” 501 of interest, but also word patterns 502, 503 in the upper and lower rows that overlap with the rectangle of “as” 501 when viewed in the vertical direction. When viewed in the horizontal direction 506, there is an overlap with the rectangle "as", and the word patterns 504 and 505 adjacent to "as" are also extracted as the neighboring word patterns together with the rectangular coordinates.

  Details of the operation of the rectangular coordinate extraction processing 302 will be described with reference to FIG.

  When the image 401 stored in the image data 181 is input to the word rectangular coordinate extraction unit 151, a binary image is first generated (601). Next, in order to detect a rough position of the character line, a black pixel is added in the horizontal direction of the image to obtain a projection distribution (602).

  FIG. 7 is a diagram showing the principle of calculation of the black pixel projection of a character line included as image information. The origin of the image is at the upper left, 702 corresponds to the coordinate axis in the image vertical direction, and 703 represents the projection distribution in the horizontal direction. After calculating the projection distribution, non-zero value ranges 704 to 708 are calculated in order to obtain the range of the vertical coordinate in which the character line exists (603). In the range of the vertical coordinate where the obtained character line exists, a connected component which is a block in which black pixels are connected is obtained and the circumscribed rectangular coordinates are also calculated. Then, the rectangles close to each other are integrated in the horizontal direction while referring to the gaps between the circumscribed rectangles of the connected components (604), and the coordinates of the integrated rectangles are stored in the word recognition result data 182 as the rectangle coordinates of the word. (605).

  FIG. 8 shows a configuration example of the word recognition result data 182.

  Reference numerals 817 to 822 denote data records of each word, 810 is a character code storage area of the word, 811 is a rectangle upper left X coordinate, 812 is a rectangle upper left Y coordinate, 813 is a rectangle lower right X coordinate, and 814 is a rectangle lower right. Y coordinate 815 represents the line number of the line containing the word. Reference numeral 816 denotes a target word flag for distinguishing between a peripheral word and a target word, and stores “1” for the target word and “0” for other peripheral words. In step 605, data 811 to 816 are stored.

  Next, the word recognition unit 152 performs a character recognition process on the English word pattern existing in the extracted rectangular coordinates with respect to the extracted English word rectangular coordinates 501 to 506. Specifically, the rectangular coordinate data 811 to 814 of the word recognition result data 182 and the image data 181 are input, and the character code of the word as the recognition result is stored in the word character code storage area 810 of the same word recognition result data 182. It outputs (303).

  The link DB search unit 160 reads the content of the word recognition result data 182 and searches the link DB 184 for link data corresponding to the English sentence including “as” (304). English sentences including specific words such as “as” are generally present in a plurality of locations in the common link DB 184. Therefore, in order to narrow down the link search target English from a plurality of English candidates, the geometrical relative relationship between “as” 501 and the English words 502 to 506 existing around “as” 501 in FIG. Use.

  FIG. 9 shows the structure of data stored in the link DB 184.

  900 is an English word index section for searching English sentences including the word, 901 is an English data storage section storing geometric information and English word character codes included in the English words, and 902 corresponds to each English sentence. The link data storage unit stores link data to be stored. The English word index unit 900 is an index table for searching the English data 901 that is the link reference object at high speed, 903 is the character code data of the word, 904 is the number of the word appearing in the document, and 905 and 906 are the word Is a pointer to the English text data storage unit 901 including

  The contents of each column of the word data records 916 to 922 constituting one English sentence in the English data storage unit 901 will be described. Reference numeral 910 denotes English word data constituting an English sentence. Reference numerals 911 and 912 denote the rectangular center-of-gravity coordinates of the English word when the upper left corner of the page containing English text is used as the origin, 911 denotes the X coordinate value, and 912 denotes the Y coordinate value. These coordinate values may be determined assuming an arbitrary resolution or an appropriate scale. Reference numerals 913 and 914 respectively denote the number of pages and the line number in which the word is included. Reference numeral 915 indicates the number of the word counted from the left in the line including the word. The record 922 includes “.” (Period) indicating the end of English text and a pointer to the link data storage unit 902. In the link data storage unit 902, reference numerals 923 to 925 indicate link data corresponding to each English sentence. A pointer to the link data storage unit 902 exists in sentence units, and a pointer is set so that one English sentence corresponds to one link data, in the case of this embodiment, a URL character string.

  Next, the details of the link reference candidate sentence selection processing operation 304 will be described with reference to FIG.

  First, the rectangular centroids of neighboring words “we” 502, “have” 503, “resulting” 504, “high” 505, and “create” 506 are extracted from the rectangular centroid of the focused word “as” 501 extracted in step 303. A line is drawn on the coordinates, and the angle or cosine of the line segment is measured (1001).

  Details of this processing will be described with reference to FIG. Lines drawn from the rectangular centroid of the word of interest “as” to the rectangular centroids of the neighboring words are indicated by 1101, 1102, 1103, 1104, and 1105. For example, an angle between line segments drawn by neighboring words “we” and “have” is indicated by 1106. Hereinafter, similarly, the angle between line segments drawn on the centroid of each neighboring word is calculated. The angle is calculated here because the scale and shooting angle of the image taken with the camera and the scale and shooting angle used to represent the position of the word when registering the dictionary are generally different. This is to extract the amount describing the geometric relative relationship. An arbitrary amount can be used as long as it corresponds to an amount that hardly changes. For example, a relative line number to which each word belongs may be used. “We” and “have” are the first line, “resulting”, “as”, and “high” are the second line, “create” the third line, and the like. Alternatively, it is also possible to use character and background color information, character size, ruled line layout, and relative arrangement relationship between ruled lines and characters.

  Next, referring to the “as” record 908 in the English index part 900 of the link DB 184, the English data containing “as” in the English data storage part 901 is searched, and whether or not there is a nearby word is also searched. (1002). Specifically, after referring to the data of the record 919 containing “as” first, among the words existing in the lines before and after the line containing “as” while referring to the row number information in the column 914. The neighborhood word obtained from the input image is searched. As a result of the search, it is determined whether the number of neighboring words is equal to or greater than a threshold (1003). As a result of the determination, if the number of neighboring words is greater than or equal to the threshold, it is considered that there are neighboring words and the process proceeds to step 1004. On the other hand, if it is not equal to or greater than the threshold, the process proceeds to step 1007. Here, the reason why the number of neighboring words is not found but is set to be equal to or greater than the threshold is that a correct word cannot be obtained due to misrecognition.

  Next, for the obtained neighborhood word, the angle between the line segments drawn from the rectangular centroid of the word of interest “as” to the rectangular centroid of the neighboring word is calculated by the same processing as in step 1001 (1004). Specifically, referring to the rectangular centroid X-coordinate value 911 and the Y-coordinate value 912 of the neighboring word, the angle between the rectangular centroids of each obtained neighboring word and the angle obtained in step 1001 Take the difference. Here, in order to reduce the calculation cost, the cosine difference of the angle is defined as the degree of difference instead of the angle difference.

For example, using the word rectangle coordinates stored in the dictionary data, the angle cosine calculation between the rectangle centroid coordinates of the words “we” and “have” with the rectangle centroid of “as” as a base point is as follows.
Rectangular centroid vector from “as” to “we”: a = (x, y) = (330-400, 225-300) = (-70, -75)
Rectangular centroid vector from “as” to “have”: b = (x, y) = (470-400, 225-300) = (+ 70, -75)
Cosine between vectors a and b = a · b / (‖a‖‖b‖) = 0.07
Next, it is determined whether or not the degree of difference is equal to or less than a threshold value (1005). In step 1006, word data after the “as” record 919 is searched in the English text storage unit 901, and a record 922 including “.” (Period) as a symbol at the end of the sentence is searched. Then, a pointer to the link data stored after the period is acquired and stored in the link reference candidate sentence search result data 183.

  The search result data 183 stores not only a pointer to link data but also all the words included in the link reference target English sentence, layout information associated with the word, and a flag value indicating the word of interest.

  FIG. 12 shows a configuration example of the link reference candidate sentence search result data 183.

  1200 is the number of words constituting the English text of the link reference candidate, and 1201 to 1205 are attribute data of each word included in the English text. Reference numeral 1201 denotes a character code storage area for words, and 1202 to 1204 respectively copy and store the data in columns 913 to 915 in FIG. 9. Each data includes the page number, the line number, and the left in the line. It is a word number counted from. Reference numeral 1205 denotes a flag representing a target word. If the target word is “1”, “1” is stored. Otherwise, “0” is stored. This flag is used for screen display control of the display unit. Reference numerals 1206 to 1213 denote word records constituting link reference English sentence candidates, and there are as many records as indicated by 1200. Reference numeral 1214 denotes a pointer to link data corresponding to the English sentence, and the pointer data of the record 922 in FIG. 9 is copied and stored.

  Steps 1002 to 1006 are repeated until there are no pointers after 905 stored in the record 908, that is, until there is no pointer to the word “as” (1007). The link data candidate pointer and English data obtained in this way are read out from the link reference candidate sentence search result data 183, and the “as” neighborhood image and rectangular coordinates are read out from the image data 181 and the word recognition result data 182, respectively. These are displayed on the display unit 120 (1008).

FIG. 13 shows an example of the display of link reference English sentence candidates.
Reference numeral 1300 denotes a line image including “as” which is a focused word of the input image, and reference numeral 1301 denotes a rectangle indicating the focused word. 1302 indicates the position where the target word “as” exists as a result of searching the link DB 184. In this example, the target word “as” exists in the third word from the left on page 24, the fifth line. ing. In addition, although an underline is displayed here, when a character with this underline is selected, a hyperlink that can display a result of accessing the corresponding link (URL) data is displayed. When this link is selected, the result of accessing the link data with reference to the pointer stored in 1214 in FIG. 12 can be displayed.

  Reference numeral 1303 is an example of English display including the word 1302, and in order to save display space, only the first and last words of the English sentence and the neighboring words before and after the word of interest are displayed here. In addition, “as” is displayed in bold to clearly indicate the word of interest. Reference numerals 1302 and 1303 are paired, and in the following, reference numerals 1304 to 1307 are displayed in pairs, in which the position of the word of interest and the English sentence including the word of interest are paired.

  As described above, the processing of steps 1001 to 1008 in FIG. 10 searches for the link data candidate corresponding to the English sentence including the target word “as”, and the result is displayed on the display unit.

  The user selects a corresponding target (1302 in this example) from among the link reference English candidates displayed in FIG. 13 by operating the input unit 130. At that time, the retrieval unit 160 retrieves the link data 924 shown in FIG. 9 with reference to the corresponding link data pointer (305), refers to the URL that is the link data, accesses the WEB server 202, and obtains the result as the link reference data. The data is stored in 185 and displayed on the display unit 120 (306).

  FIG. 14 shows a display example of the access result of the link data.

  1400 and 1401 are the same as 1300 and 1301 in FIG. 13, and 1402 is the result of accessing link data corresponding to the English text of 1303. After referring to the pointer 1214 in FIG. 12, 924 in the link DB 184 (FIG. The result of obtaining the link data of 9) and accessing the URL is displayed. Here, assuming that the image data of “Product manufacture Process” is stored in the accessed URL, the data is displayed on the screen.

  As described above, the process shown in FIG. 3 allows a user to capture a part of an English sentence to specify where the photograph is photographed and to obtain an access result of link data of the corresponding English sentence. it can.

  Next, a second embodiment will be described with reference to FIGS.

  In the second embodiment, as shown in FIG. 26, the above-described link DB and its search means are held in the link database (DB) server 203, and data is transmitted from the link DB server 203 via the communication network 201 as necessary. Is downloaded to the portable information terminal 100. Reference numeral 202 denotes a WEB server as in FIG. That is, the portable information terminal 100 performs word recognition from the acquired image, transmits the recognition result to the server 202, searches the link DB server 203 using the character code that is the recognition result, and retrieves the search result. Will be described below with reference to a system configuration and processing procedure for returning the message to the portable information terminal 100 for display.

  FIG. 15 is a block diagram of a portable information terminal according to the second embodiment.

  This example is also an example in which link information associated with an English sentence is acquired as in the first embodiment. The main difference from the configuration of FIG. 1 is that a storage unit for link data 186 that stores link data itself is provided instead of the link DB 184, and the processing procedure by the character recognition / link DB search control unit 145 changes. Etc.

  FIG. 16 shows a schematic configuration of the link DB server 203.

  In the link DB server 203, a communication control unit 1601, a central processing unit 1602, and a memory 1603 are connected via a bus 1600. The memory 1603 stores a link DB search program 1604. This program is a program for performing link DB search processing in the same manner as the link DB search unit 160 in FIG. The word recognition result data 1605, the link reference candidate sentence search result data 1606, and the link DB 1607 are areas for storing data equivalent to the word recognition result data 182, the link reference candidate sentence search result data 183, and the link DB 184 in FIG. 1, respectively. .

  FIG. 17 is a flowchart showing respective processes of the portable information terminal 100 and the link DB server 203 in the second embodiment.

  In the figure, solid arrows indicate the flow of processing, and dotted arrows indicate the flow of data.

  The image input process (1701) to the English word recognition process (1703) are the same as steps 301 to 303 in FIG. 3, and the English word from the image partially including the link reference target sentence photographed by the image photographing unit 110 is used. And the recognition result is stored in the word recognition result data 182. Next, the character recognition / link DB search control unit 145 transmits the English word recognition result data stored in the word recognition result data 182 to the link DB server 203 via the communication control unit 141 (1704).

  The link DB server 203 receives the word recognition result data transmitted from the portable information terminal 100 and stores the data in the word recognition result data 1605 (1710). Thereafter, the link DB search program 1604 is executed to search the link DB 1607 to search for a link reference candidate sentence (1711). This process is a process of searching the link DB 1607 using the word recognition result data 1605 in the same processing procedure as steps 1001 to 1007 in FIG. 10 and storing the search result in the link reference candidate sentence search result data 1606 of the memory 1603. is there.

  Next, the server 202 transmits the link reference candidate sentence search result data 1606 stored in the memory 1603 to the portable information terminal 100 (1712), and the portable information terminal 100 receives the data to search for the link reference candidate sentence search. The result data 183 is stored (1705).

  The portable information terminal 100 displays the link reference candidate search result as shown in FIG. 13 on the display unit 120 with reference to the link reference candidate sentence search result data 183. The user selects one of the link candidates displayed on the display unit (1706). As a selection result, only the data of the pointer 1214 to the link data corresponding to the English text in FIG. 12 is transmitted to the server 202 (1707).

  The link DB server 203 receives a pointer to the link data transmitted from the portable information terminal 100 (1713), and searches for the corresponding link data with reference to the received pointer (1714). Then, link data obtained as a result of the search is transmitted to the portable information terminal 100 (1715). The portable information terminal 100 receives the retrieved link data, stores it in the link data 186 of the memory 180 (1708), refers to the URL that is the link data, accesses the WEB server 203, and links the reference data. The data is stored in the reference data 185 and displayed on the display unit 120 (1709). The contents of the display screen of the display unit 120 are the same as those in FIG.

  As described above, the processing operation shown in FIG. 17 eliminates the need for the user to shoot the entire English text as a link reference, and uses the link DB 1607 provided in the link DB server 203 and the search means for searching the DB. The target link data can be obtained via the network. In the first and second embodiments, the link data storage unit 902 in the link DB 184 and the link DB 1607 stores a URL character string as link information, but instead of link information, actually referred data such as an image is stored. It may be stored.

  Next, a third embodiment will be described with reference to FIGS.

  In this example, link data embedded in a Japanese sentence is referred to. In the case of English sentences, there is a space between words (in the case of the first and second embodiments). However, in the case of the Japanese sentence in the present embodiment, the Japanese sentence to be character-recognized is a sentence without a space between words. In the case of Japanese sentences, an English word corresponds to a “morpheme” that is a component of the Japanese sentence, and is characterized by character recognition for “morpheme” and link candidate sentence search using the same. Hereinafter, description will be given with reference to the drawings.

  FIG. 18 is a block diagram showing the configuration of the portable information terminal.

  The main difference from the configuration of the portable information terminal 100 shown in FIG. 1 is the configuration and processing operation of the character recognition unit 150. That is, the character line coordinate extraction unit 1501 and the morpheme recognition unit 1502 of the character recognition unit 150 of the morphological information terminal 100 shown in FIG. 18 correspond to the word rectangular coordinate extraction unit 151 and the word recognition unit 152 of FIG. In the memory 180, character line coordinate data 1804 is newly added. The morpheme recognition result data 1805, the link reference candidate sentence search result data 1806, the link DB 1807, and the link reference data 1808 respectively correspond to 182, 183, 184, and 185 in FIG.

  FIG. 19 shows the link processing flow of the third embodiment. Hereinafter, the processing operation of the portable information terminal of FIG. 18 will be described with reference to FIG.

  First, a character recognition target image photographed by the image photographing unit 110 is input (1901).

  FIG. 20 shows an example of a Japanese document to be recognized. There is a display of “P24” and page number in the upper right. Reference numeral 2001 denotes a range of an image captured and captured by the image capturing unit 110, and an image in this range is input to the character recognition unit 150. In this example, an image is used to access the link data attached to the second sentence that begins with "Respond to this ..." on the third line and ends with "..." on the fifth line. Assuming that In order to extract link information from this document, in this embodiment, any morpheme constituting the second Japanese sentence can be the target of attention. Here, it is assumed that the “+” mark indicating the target of interest overlaps the “product” in the fifth row.

  The character line extraction unit 1501 of the character recognition unit 150 extracts rectangular coordinates indicating the location of the character line from the input image (1902). This process is the same as step 302 in FIG. Here, not only the character line coordinates including the “product” of interest but also the upper and lower character line coordinates are output together. In step 302, rectangular coordinates in units of words are extracted. However, in the case of Japanese, there is no blank corresponding to a word break. Therefore, after calculating the position of the line by taking the black pixel projection in the line direction as shown in FIG. 7, the rectangular coordinates of the character line are calculated by integrating the adjacent connected components included therein, and the character line Stored in the coordinate data 1804.

  The morpheme recognition unit 1502 performs character recognition processing on the character pattern existing in the extracted rectangular coordinates and performs morphological analysis on the extracted rectangular coordinates of the character line, and the character code of each morpheme is obtained as a character recognition result. The center of gravity coordinates of the rectangle corresponding to the morpheme are output to the morpheme recognition result data 1805 (1903). Here, as shown in FIG. 21, the morpheme patterns of the upper and lower character lines that overlap in the horizontal direction with the rectangular coordinates of “product” 2101, and the character codes and the center of gravity of the rectangles 2102 to 2109 of “product” adjacent morpheme patterns Coordinates are extracted.

  FIG. 22 shows the configuration of morpheme recognition result data 1805.

  Reference numerals 2216 to 2224 denote data records of each morpheme. Reference numeral 2210 denotes a morpheme character code storage area, and reference numerals 2211 to 2214 denote the upper left X coordinate, upper left Y coordinate, lower right X coordinate, and lower right Y coordinate of the morpheme rectangle, respectively. Reference numeral 2215 denotes the line number of the line containing the morpheme. Reference numeral 2216 denotes a focused morpheme flag for distinguishing the peripheral morpheme from the focused morpheme, and stores “1” for the focused morpheme and “0” for other peripheral morphemes.

  The character recognition result is passed to the link DB 1807, and a link data candidate corresponding to the Japanese sentence including “product” is searched for using the character recognition result as a key by the same method as the processing described in the first embodiment (1904).

  FIG. 23 shows the structure of data stored in the link DB 1807. Reference numeral 2300 denotes a morpheme index unit for searching a Japanese sentence including the morpheme, 2301 denotes a Japanese sentence data storage unit, and 2302 denotes a link data storage unit. In the morpheme index unit 2300, 2303 is morpheme data, 2304 is the number of the morpheme appearing in the document, and 2305 and 2306 are pointers to the Japanese sentence data storage unit 2301 containing the morpheme.

  The contents of each column of records 2316 to 2322 of morpheme data constituting one Japanese sentence in the Japanese sentence data storage unit 2301 will be described. Reference numeral 2310 denotes morpheme data constituting a Japanese sentence. Reference numerals 2311 and 2312 denote the rectangular center-of-gravity coordinates of the morpheme when the upper left of the page including the Japanese sentence is the origin, 2311 denotes the value of the X coordinate, and 2312 denotes the value of the Y coordinate. Reference numerals 2313 and 2314 respectively denote the number of pages and the line number including the morpheme. Reference numeral 2315 denotes the number of the morpheme counted from the left in the row including the morpheme. A record 2322 includes “.” (Reading mark) indicating the end of the Japanese sentence and a pointer to the link data storage unit 2302. In the link data storage unit 2302, 2323 to 2325 indicate link data corresponding to each Japanese sentence.

  In step 1904, the link data pointer and the corresponding Japanese sentence data are retrieved from the link DB using the relative arrangement information between the morphemes as in the first embodiment and displayed on the display unit. Select a link data pointer of a possible reference candidate.

  Details of step 1904 will be described with reference to FIG.

  First, a line is drawn from the rectangular center of gravity of the target morpheme “product” extracted in step 1903 to each rectangular barrel coordinate from the neighboring morpheme 2102 to 2109 shown in FIG. 21, and the angle between the line segments is the same as in step 1001 of FIG. Alternatively, the cosine is measured (2401).

  Next, with reference to the “product” record 2308 of the morpheme index part 2300 of the link DB 1807, the Japanese sentence data containing “product” in the Japanese sentence data storage part 2301 is searched, and whether or not the neighboring morpheme exists is determined in step 1002. The search is performed by the same processing procedure as (2402). As a result of the search, it is determined whether or not the number of neighboring morphemes is equal to or greater than a threshold value. If it is equal to or greater than the threshold value, it is considered that there is a neighboring morpheme, and the process proceeds to Step 2404. On the other hand, if it is not more than a threshold value, it will progress to step 2407 (2403).

  If the number of neighboring morphemes is equal to or greater than the threshold, the angle of the line segment drawn from the rectangular centroid of the focused morpheme “product” to the rectangular centroid of the neighboring morpheme is calculated for the obtained neighboring morpheme by the same process as in step 2401 The degree of difference is calculated (2404). This calculation is the same calculation process as in step 1004. Thereafter, it is determined whether or not the degree of difference is equal to or less than a threshold (2405). If the difference is equal to or less than the threshold, the process proceeds to step 2406, and if it exceeds the threshold, the process proceeds to step 2407.

  If the degree of difference is less than or equal to the threshold value, the Japanese data storage unit 2301 searches for word data after the “product” record 2319, and searches for a record 2322 that includes the symbol “.” (Reading mark) at the end of the sentence. Then, a pointer to the link data stored after the punctuation mark is acquired and stored in the link reference candidate sentence search result data 1806 (2406).

  FIG. 25 shows a configuration example of the link reference candidate sentence search result data 1806.

  Reference numeral 2501 denotes the number of morphemes constituting the Japanese sentence of the link reference candidate, and reference numerals 2502 to 2506 denote attribute data of each morpheme included in the Japanese sentence. Reference numeral 2502 denotes a morpheme character code storage area, and reference numerals 2503 to 2505 respectively copy and store the data in columns 2313 to 2315 in FIG. 23. Each data has a page number, a line number, and a left in the line. It is a morpheme number counted from. Reference numeral 2306 denotes a flag indicating the target morpheme. If the target morpheme, “1” is stored, and “0” is stored otherwise. This flag is used for screen display control of the display unit.

  Reference numerals 2507 to 2513 denote morpheme records constituting a Japanese sentence of link reference candidates, and there are as many records as indicated by 2501. Reference numeral 2514 denotes a pointer to link data corresponding to the Japanese sentence, and the pointer data of the record 2322 in FIG. 23 is copied and stored.

  The processing from step 2402 to 2406 is repeated until there is no pointer after 2305 stored in the record 2308, that is, until there is no pointer to the morpheme “product” (2407). The link data candidate pointer and Japanese sentence data obtained in this way are read out from the link reference candidate sentence search result data 1806, and the “product” neighborhood image and rectangular coordinates are read out from the image data 181 and the morpheme recognition result data 1804, respectively. These are displayed on the display unit 120 (2408).

  As described above, the link data candidate corresponding to the Japanese sentence including the target morpheme “product” is searched by the process shown in FIG. 24, and the result is displayed on the display unit. When the user selects a pointer related to the relevant link data among the reference candidate link data, the corresponding link data is searched from the link DB 1807 (1905). Then, the WEB server 202 is accessed with reference to the URL as link data, and the result is stored in the link reference data 1808 and displayed on the display unit 120 (1906).

  As described above, the process shown in FIG. 19 allows the user to capture a part of the Japanese sentence, identify where the photograph is taken, and obtain the access result of the corresponding Japanese link data. .

  In the above embodiment, the morpheme is used as a unit. However, it is only necessary to know which link reference target Japanese sentence is indicated by the “+” mark on the display unit. Therefore, in an alternative example, a character string set to an arbitrary character length, for example, one character or two characters, may be processed instead of a morpheme without performing character recognition or search processing in units of morphemes.

  Furthermore, another modified example will be described. In the link DB of FIG. 9, a pointer to link data is provided for each sentence. However, this does not have to be in sentence units, and it is possible to provide pointers to link data in word units. FIGS. 27 and 28 show the changed portions when the pointer is stored in units of words, and corresponds to the link DB of FIG. 9 and the reference target candidate sentence search result data of FIG. 12, respectively. A column 930 is a pointer to the link data storage unit 902 stored in 922 in FIG. 9, and a pointer is stored for each word as indicated by 931 to 938. Reference numerals 939 to 941 denote link information associated with each word. In FIG. 28, pointers such as 1220 are inserted on the right side of the column 1205 for each word. For example, the pointers in the columns 1220 to 1226 are copied from the pointers stored in the columns 930 to 935 in FIG. And save it. In this embodiment, since the flag “1” is set in the column 1205 of the 1225 record as the word of interest, 940 link data is acquired by referring to the pointer of the 1225 record. By providing a similar configuration, in addition to word units, in the case of Japanese sentences, it is possible to associate link data with morpheme units or character units. Furthermore, it is also possible to provide a pointer to link data in any unit such as a section unit or a chapter unit in addition to a sentence unit.

The figure which shows the structure of the document link system using the portable information terminal by 1st Embodiment. The figure which shows the document link system containing the portable information terminal and WEB server by 1st Embodiment. The figure which shows the link data reference processing flow in 1st Embodiment. The figure which shows the example of the English of the link reference object in 1st Embodiment. The figure which shows the example of extraction of the word rectangular coordinate in 1st Embodiment. The figure which shows the processing flow of the English-language rectangle coordinate extraction 302 in 1st Embodiment. The figure for demonstrating the principle of the black pixel projection process for the English word rectangular coordinate extraction 302 in 1st Embodiment. The figure which shows the structural example of the word recognition result data 182 in 1st Embodiment. The figure which shows the structure of link DB184 which stores the link data linked | related with the English sentence in 1st Embodiment. The figure which shows the link DB search processing flow for searching the link information linked | related with the English sentence in 1st Embodiment. Explanatory drawing of the angle measurement between surrounding words on the basis of the attention word in 1st Embodiment. The figure which shows the structural example of the reference target candidate sentence search result data of the English sentence in 1st Embodiment. The figure which shows the example of a display of the link reference object English sentence candidate in 1st Embodiment. The figure which shows the example of a display of the reference result of the link data in 1st Embodiment. The figure which shows the structure of the portable information terminal by 2nd Embodiment. The figure which shows the structure of the link DB server by 2nd Embodiment. The figure which shows the link data reference processing flow in 2nd Embodiment. The figure which shows the structure of the portable information terminal by 3rd Embodiment. The figure which shows the link data reference processing flow in 3rd Embodiment. The figure which shows the example of the Japanese sentence of the character recognition and link reference object in 3rd Embodiment. The figure which shows the example of extraction of the morpheme rectangular coordinate in 3rd Embodiment. The figure which shows the structural example of the morpheme recognition result data in 3rd Embodiment. The figure which shows the structure of link DB1807 which stores the link data linked | related with the Japanese sentence in 3rd Embodiment. The figure which shows the search processing flow of link DB for searching the link data linked | related with the Japanese sentence in 3rd Embodiment. The figure which shows the structural example of the link reference candidate sentence search result data 1806 of the Japanese sentence in 3rd Embodiment. The figure which shows the document link system containing the portable information terminal by 2nd Embodiment, a WEB server, and a link database server. The figure which shows the structure of link DB184 which stores the link data linked | related with the English word in 1st Embodiment. The figure which shows the structural example of the reference object candidate sentence search result data which stored the pointer of the word unit in 1st Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Portable information terminal 110 ... Image pick-up part 120 ... Display part 130 ... Key input part 140 ... Terminal control part 150 ... Character recognition part 160 ... Link DB search unit, 180 ... memory, 184 ... link DB, 202 ... server

Claims (15)

Using an image capturing unit, an input unit that is operated by a user to input information, and a portable information terminal having a display unit, link information associated with characters included in the image captured by the image capturing unit is displayed. In the document link information acquisition system that displays the link destination data on the display unit with reference to the link information,
A storage device that stores a document link information database that holds a sentence composed of words and link information related to the sentence in association with each other, and associates and holds arrangement information of each word included in the sentence;
A character recognition unit that performs character recognition processing on a target word pattern designated by a user from an image captured by the image capturing unit and a word pattern around the target word pattern, and outputs a character code of the word pattern When,
An extraction unit that extracts relative arrangement information regarding a target word pattern designated by a user and a word pattern around the target word pattern;
The document link information database using the character code obtained from the character recognition unit and the relative arrangement information obtained from the extraction unit with respect to the word pattern of interest and a word pattern around the word pattern of interest. A search section for searching for,
A link information acquisition system for displaying link destination data on the display unit with reference to link information obtained from the search unit.   2. The document link information acquisition system according to claim 1, wherein the document link information database unit stores an English sentence and its link data in association with each other.   The document link information acquisition system according to claim 1, wherein the document link information database unit stores a sentence composed of words separated by spaces and link information related to the sentence in association with each other.   The relative arrangement information is an angle of a line segment drawn from a rectangular centroid of a target word pattern specified by a user to a rectangular centroid of each peripheral word pattern of the target word pattern. Item 1. The document link information acquisition system according to Item 1.   The relative arrangement information is relative arrangement information of a line including a target word pattern specified by a user and a line including each peripheral word pattern of the target word pattern. Item 1. The document link information acquisition system according to Item 1. An image capturing unit; an input unit that is operated by a user to input information; and a link information acquiring unit that performs processing for obtaining link information associated with characters included in an image captured by the image capturing unit; In a document link information acquisition system using a portable information terminal having a display unit that displays link destination data with reference to link information obtained by the link information acquisition unit,
In addition to storing link information related to each sentence in the document,
A storage device that stores a document link information database that holds and associates placement information of each morpheme included in each sentence;
A character recognition unit that performs character recognition processing on a morpheme pattern specified by the user from an image captured by the image capturing unit and a morpheme pattern around the morpheme pattern of interest, and outputs a character code of each morpheme;
An extraction unit that extracts relative arrangement information regarding a morpheme pattern of interest and a morpheme pattern existing around the morpheme pattern of interest designated by the user;
The document link information using the character code obtained from the character recognition unit and the relative arrangement information obtained from the extraction unit for the target morpheme pattern and the morpheme pattern existing around the target morpheme pattern A search section for searching a database;
A link information acquisition system for displaying link destination data on the display unit with reference to link information obtained from the search unit. Using an image capturing unit, an input unit that is operated by a user to input information, and a portable information terminal having a display unit, link information associated with characters included in the image captured by the image capturing unit is displayed. In the document link information acquisition system that displays the link destination data on the display unit with reference to the link information,
A storage device that stores a document link information database that holds a sentence composed of words and link information related to the sentence in association with each other, and holds association information of each character included in the sentence, and
A character recognition unit that performs character recognition processing on a target character pattern designated by a user from an image captured by the image capturing unit and a character pattern around the target character pattern, and outputs a character code of the character pattern When,
An extraction unit that extracts relative arrangement information regarding the character pattern of interest specified by the user and the character patterns around the character pattern of interest;
The document link information database using the character code obtained from the character recognition unit and the relative arrangement information obtained from the extraction unit with respect to the character pattern around and the character pattern around the character pattern of interest. A search section for searching for,
A link information acquisition system for displaying link destination data on the display unit with reference to link information obtained from the search unit. A document link information acquisition system having a storage device for storing the document link information database and a server including the search unit,
The character code obtained from the character recognition unit of the portable information terminal and the relative arrangement information obtained from the extraction unit are transmitted to the server via a network, and the search unit in the server 8. The document link information acquisition system according to claim 1, wherein link information obtained as a result of searching the document link information database is transmitted to the portable information terminal via the network. A document link information acquisition system including a server including the character recognition unit, the extraction unit, a storage device that stores the document link information database, and the search unit,
An image obtained from the image photographing unit of the portable information terminal is transmitted to the server via a network, and link information obtained as a result of retrieving the document link information database by the retrieval unit in the server is used as the network. The document link information acquisition system according to claim 1, wherein the document link information acquisition system transmits the document link information to the portable information terminal.   The document link information database includes a word index unit for searching for a sentence including a word, a sentence data storage unit storing geometric information of a word included in the sentence and a character code of the word, and a relation to each sentence The document link information acquisition system according to claim 1, further comprising a link information storage unit that stores link information to be stored.   A portable information terminal used in the system according to claim 1.   The server used in the system of Claim 8 or 9.   The system according to claim 1, wherein the program is executable on a computer for realizing the function of the extraction unit. Using an image capturing unit, an input unit that is operated by a user to input information, and a portable information terminal having a display unit, link information associated with characters included in the image captured by the image capturing unit is displayed. In the document link information acquisition method of referring to the link information and displaying the link destination data on the display unit,
Preparing in advance in a storage device a document link information database that holds a sentence composed of words and link information related to the sentence in association with each other, and holds the arrangement information of each character included in the sentence in association with each other;
Performing character recognition processing on a target character pattern designated by a user from an image captured by the image capturing unit and a character pattern around the target character pattern, and outputting a character code of the character pattern;
Extracting relative arrangement information regarding the character pattern of interest specified by the user and the character patterns around the character pattern of interest;
The document link information database using the character code obtained from the character recognition unit and the relative arrangement information obtained from the extraction unit with respect to the character pattern around and the character pattern around the character pattern of interest. Searching for
Displaying link destination data on the display unit with reference to link information obtained as a result of the search; and
Document link information acquisition method comprising: As a result of searching the document link information database, displaying a sentence that is one or more link information candidates including the target character pattern on the display unit;
In the case of a plurality of candidates, referring to link information corresponding to a candidate designated by the user via the input unit from among the plurality of candidates, obtaining link destination data from the document link information database; Further comprising
15. The document link information acquisition method according to claim 14, wherein the link destination data obtained from the document link information database and the sentence are displayed in correspondence with each other on the display unit.

Images