JP3917349B2 - Retrieval device and method for retrieving information using character recognition result - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a search apparatus and a method for searching for information in an image by using a result of character recognition processing on image information.
[0002]
[Prior art]
2. Description of the Related Art In recent years, document management systems that share documents electronically have been used in organizations such as companies from the viewpoint of information sharing and rapid information use. As a computerized document, a text searchable document such as a word processor document or a presentation document that has been digitized from the time of creation is often used. In addition, image information that cannot be text-searched as it is is also used.
[0003]
As a method of registering an image in a document management system, a method of extracting text information that can be searched by character recognition of the image and storing the information together with the image is generally used for later search processing.
[0004]
However, the recognition rate of the character recognition process is not 100%, and the recognition result includes erroneous recognition. Further, the normal search method cannot detect misrecognition. For this reason, even if there is a keyword character string in the document, the character string does not match the keyword due to misrecognition, and search may not be possible.
[0005]
In order to prevent this, it is necessary to develop a search data creation method using character recognition characteristics and a search method for data after character recognition. Examples of conventional search techniques for searching for information after character recognition include the following documents.
(# 1) Marukawa, Fujisawa, Shima “Study of information retrieval method that allows output ambiguity of recognition function Analysis and evaluation of retrieval method focusing on recognition error characteristics” IEICE Journal vol.J79-D-II, no.5, pp785-794, 1996
(# 2) Ohta, Takasu, Adachi “Full-text search method in Japanese text including recognition errors”, Journal of the Institute of Information Processing, vol.39, no.3, pp625-635, 1998
(# 3) Itonori, Ozaki “Japanese word extraction with similar characters”, Technical Report of IEICE PRMU 98-87, pp25-32, 1998
(# 4) Imagawa, Matsukawa, Kondo, Mekata “Search method from text including misrecognition with recognition reliability for each character” IEICE Technical Report PRMU99- 72, pp63-75, 1999
(# 5) Kondo, Matsukawa, Imagawa, Eyepiece "A Study on Fuzzy Search from Texts Containing Character Recognition Errors" IEICE Tech. PRMU99-73, pp69-75, 1999
(# 6) Yusa, Tanaka “Realization of a character string search function for Japanese document images” Information processing, Information media research report 19-1, pp1-8, 1995
(# 7) Nakanishi, Omachi, Aso “Highly accurate keyword search system for low-quality document images” IEICE Tech. Bulletin PRMU98-232, pp97-104, 1999
(# 8) Matsukawa, Imakawa, Kondo, Eyepiece "Improvement of document image search performance using shape feature search" IEICE Tech. Bulletin PRMU99-74, pp77-83, 1999
These conventional techniques can be classified as follows.
a) Method for collating character recognition candidates with keywords
A recognition candidate is generated at the time of character recognition, and the search is performed while considering a plurality of possibilities using the candidate at the time of search.
Reference (# 1)
The number of recognition candidates is not a fixed number, and threshold processing of similarity is performed for each character to narrow down recognition candidates (threshold values: 4 types).
b) Keyword expansion method using similar characters
b-1) Keyword character string expansion by similar character table
One keyword is expanded and searched by similar character information.
Reference (# 1)
Simply expand keywords using a pre-made confusion matrix. The number of similar characters for one character is not a fixed number, and a threshold value process for similarity is performed for each character in advance to narrow down similar characters (threshold value: 4 types).
Reference (# 2)
A table storing probabilities based on the probability of error, which is a probability representation of misrecognition, error omission, error insertion, error coupling, and error division, is created in advance, and all non-zero probabilities are used using the table. Expand keywords in combination. Next, a full text search is performed using the expanded keywords. Then, for each character string extracted by the search, the certainty factor of the character string obtained probabilistically is calculated, and the suitability is determined using a threshold value.
b-2) Search by similar character table
A character string that has been character-recognized is searched using a similar character table in consideration of a plurality of correspondence relationships for each search.
Reference (# 3)
A plurality of characters having similar shapes are handled as one category, and search is performed by narrowing down to one by language processing.
[0006]
Dictionary creation)
First, similar character categories (clusters) are created by hierarchical clustering. Next, the similar character cluster is divided and the divided clusters are reintegrated.
[0007]
identification)
The image is processed in the order of similar character identification, morphological analysis, and detailed identification to generate a word string. In similar character identification, 1000 or 2000 categories are used.
[0008]
First, a similar character category closest to the input character is obtained by hierarchical identification of reintegrated cluster → divided cluster → similar character cluster, and a character string made of the similar character category is created. Next, words using characters in the similar character category are generated by morphological analysis for the similar character category. At this time, only characters that cannot be narrowed down to one word by morphological analysis are determined by detailed identification.
Reference (# 4)
The number of characters developed in the similar character table is variable for each character in accordance with the certainty (reliability) of the recognition result. In advance, a similar character table including correct characters, recognition results, and certainty factors is created, and character recognition results and certainty factors are output at the time of recognition. At the time of search, a matching probability with the search character is obtained from the character of the keyword, the character of the recognition result, and its certainty using a similar character table. When the value is positive, the recognition result is regarded as being collated, and the search is performed. This method is the same as the search limited to similar characters having a positive matching probability.
Reference (# 5)
In addition to the search method of the literature (# 4) using the reliability of character recognition given to each character and the probability of specifying by number of characters, when k characters of n words match, the word is uniquely specified Search using the probability that you can. The probability that can be specified by the number of characters is a probability that another word is not searched even if a part of a word having a relatively long character string length is treated as a wild card.
c) Image search method
A keyword image is directly retrieved from a document image by image matching.
Reference (# 6)
First, a character rectangle is regarded as a square and divided into 16 equal parts, and a rectangle formed by combining two adjacent portions vertically or horizontally is taken as one unit. And the combination which makes a square is extracted from the combination of two adjacent rectangles. From one character, 18 sets can be combined. Next, the black pixel density in one unit is obtained, and the ratio of the black pixel density between adjacent units is coded with 2 bits of 00 to 11 as the feature amount. A feature amount of 36 bits per character is obtained, and an error of 8 bits is allowed at the time of collation.
Reference (# 7)
A plurality of cutout candidates are output, and a feature vector is registered for each cutout candidate. At the time of retrieval, matching is performed for each character from the first character of the keyword characters, and if the distance value is equal to or less than a fixed threshold value, the match is assumed. At this time, if the previous character matches, the next character is narrowed down and the processing speed is increased. Then, the keyword is detected when the distance values of all characters are equal to or smaller than the threshold value and the sum of the distance values is equal to or smaller than another fixed threshold value.
d) A combination of similar character search and image search
Reference: (# 8)
The similar character search and the image search are switched according to the certainty (reliability) of the recognition result. The similar character search of the document (# 4) in b-2) is performed, and characters that match the matching conditions are extracted. A shape feature search is performed when characters that are not detected by the similar character search and the reliability is lower than the fixed threshold continue. In the shape feature search, adjacent low-reliability characters are integrated and cut out so as to have a width estimated from the keyword characters. Next, the character feature amount is obtained, and the Euclidean distance between the character feature amount and the keyword character feature amount is obtained. If the distance value is equal to or smaller than the fixed threshold value, it is assumed that the collation has been performed.
[0009]
[Problems to be solved by the invention]
However, the conventional search technique described above has the following problems.
a) Method for collating character recognition candidates with keywords
Reference (# 1)
Depending on the font of the search target document image, the correct character may not be included in the candidate characters. In particular, when there are a lot of similar characters, if the correct answer falls below the candidate character, it is cut off from the limitation on the number of candidate characters, and the correct character does not enter the candidate character.
b) Keyword expansion method using similar characters
b-1)
Reference (# 1)
The number of expanded keywords increases exponentially depending on the size of the similar character table, and the amount of calculation becomes enormous.
Reference (# 2)
As processing targets expand, the number of expanded keywords increases explosively.
b-2)
Reference (# 3)
Since language processing is included, the accuracy is poor for a compound word or a word with a small number of characters.
Reference (# 4)
Depending on how the similar character table prepared in advance is made, accuracy cannot be expected. In fact, it has been reported by the authors of this document that the accuracy of low resolution characters is poor.
Reference (# 5)
Weak to compound words and words with few characters. In addition, as in the method of the document (# 4), accuracy cannot be expected depending on how to prepare a similar character table prepared in advance.
c) Image search method
Reference (# 6)
With the feature values used in this method, high accuracy cannot be expected. Also, the amount of calculation is large compared to a search using only text. Furthermore, the accuracy is not high except for dictionary fonts.
Reference (# 7)
Compared to a search that uses only text, it is computationally intensive. If the first character of the keyword cannot be detected, the corresponding character string cannot be searched. Furthermore, the accuracy is not high except for dictionary fonts.
d) A combination of similar character search and image search
Reference (# 8)
Similar to the method of literature (# 4), accuracy cannot be expected depending on how to prepare a similar character table prepared in advance.
[0010]
An object of the present invention is to provide a search apparatus and method for performing a more accurate search by leaving the good part of the search technique described above and improving the incomplete part in the search process of information after character recognition. That is.
[0011]
[Means for Solving the Problems]
FIG. 1 is a principle diagram of a search apparatus according to the present invention. 1 includes an input unit 101, an extraction unit 102, an estimation unit 103, and an output unit 104, and searches for information in the document image using information on the recognition result of the document image.
[0012]
The input unit 101 inputs a keyword, and the extraction unit 102 extracts an arbitrary character constituting the keyword from the document image. Search for the searched character As key letter , Position information of the key character Extract. The estimation means 103 uses the position information of the extracted key character as a reference, and the number of characters that are on the head side of the key character among the characters constituting the keyword and the character in the document image Characters indicating typical sizes Size information and Representative Character spacing Character spacing indicating Using at least one of the information, calculate the tip position of the area corresponding to the keyword in the document image, the number of characters that are on the end side of the key character among the characters that constitute the keyword, font size Information and Character spacing The terminal position of the region corresponding to the keyword is calculated using at least one of the information, and the range between the obtained tip position and the terminal position is estimated as the region corresponding to the keyword. The output means 104 outputs the estimated area. Among them, information on the area containing the character string corresponding to the keyword ,search results As Output.
[0013]
For example, the input unit 101 inputs a keyword designated by the user to the search device, and the extraction unit 102 extracts an arbitrary character included in the input keyword from the document image and estimates it as a key character. 103. The estimation means 103 estimates an area including the key character and corresponding to the number of characters constituting the keyword from the position of the received key character. Then, the output unit 104 outputs the position and range information of the region including the character string corresponding to the keyword among the estimated regions.
[0014]
According to such a search device, it is possible to automatically estimate a keyword region on the basis of the position of a pattern corresponding to an arbitrary character constituting a keyword among character patterns included in a document image. Therefore, even if the first character of the keyword is not detected, if other characters are extracted as key characters, information on the area corresponding to the keyword is output. This improves the accuracy of the search process.
[0015]
For example, the input unit 101 in FIG. 1 corresponds to the input device 113 in FIG. 14 described later, and the extraction unit 102 and the estimation unit 103 in FIG. 1 correspond to the combination of the CPU 111 and the memory 112 in FIG. The output means 104 corresponds to the output device 114 in FIG.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The search apparatus according to the present embodiment first performs area identification and character recognition of input image information, and generates search information including a recognition result necessary for the search. Next, using the search information, the characters constituting the keyword are searched in the input image, and the range of the region where the keyword exists is estimated from the position of the characters. Then, the character string is verified using the image information in the area, and if a keyword is extracted, the position is output as a search result.
[0017]
FIG. 2 is a flowchart of the search information generation process. The search device first inputs a document image to be searched (step S1). At this time, the user optically reads a paper document with an image input device such as a scanner and inputs the paper document to the search device. Thereby, the document image is registered in the search device. As a document image, a monochrome binary image is basically assumed. However, even a color image can be handled in the same manner as a binary image by performing binarization as preprocessing.
[0018]
Next, the search device scans the document image, performs region identification by a known technique, and extracts a character region (step S2). Here, for example, the entire image is labeled to extract a circumscribed rectangle of the black pixel connection region, and a rectangle of a certain size or more is extracted as a figure / table region candidate. Next, ruled line extraction is performed for the inside of each rectangle of the figure / table area candidate. If vertical and horizontal ruled lines are extracted, the rectangle is identified as a table area. If no ruled line is extracted, the rectangle is identified as a figure region. Then, an area other than the figure / table area is extracted as a character area.
[0019]
Next, a line (character string) is extracted from the character area, character recognition is performed for each line, and search information including a recognition result is generated and output (step S3). At this time, an area of one character is determined based on a circumscribed rectangle of the black pixel connection area, and a character pattern is cut out from the image information.
[0020]
However, since the characters may be in contact with each other, the contact characters are divided using the contact character cut-out technique, and the search information is output in units of characters or character partial patterns (basic rectangles). As the contact character segmentation technique, for example, the technique of Japanese Patent Application Laid-Open No. 11-338975 (“a recording medium on which a character segmentation processing method and a character segmentation processing program are recorded”) is used.
[0021]
In addition, in order to accurately extract a separation character such as “river”, adjacent basic rectangles are integrated to create an integrated rectangle of character candidates, and similar search information is output for the rectangle.
[0022]
Hereinafter, the basic rectangle that is the output unit of the search information and the integrated rectangle are collectively referred to as a character rectangle. The search information is output line by line, and the information of each character rectangle is sorted and output within one line. The search information for each character rectangle includes the following.
a) Character recognition result information
This information is information of recognition candidates from the first position to the nth position obtained by character recognition (n ≧ 1). The content of each candidate information is a character code, a distance value, an evaluation value, or a certainty factor. All character codes are converted to full-width codes for searching. The distance value represents the similarity between the character pattern to be recognized and the character pattern in the recognition dictionary, and the evaluation value represents the evaluation for the recognition candidate. The certainty factor represents the likelihood of the recognition candidate, and is calculated using, for example, the distance value of the candidate and the distance value of another candidate.
b) Character graphic feature information
This information is information representing the graphic characteristics of a plurality of characters, and includes the following information.
・ Character pattern itself
・ Character pattern information (image features)
Information with a clear character recognition capability is desirable. Specifically, a feature vector or the like is used. When the feature vector in the recognition dictionary is matched with this feature vector using the character code of one character in the keyword as a key, the same character can be reliably matched, but different characters cannot be matched. Is done. For example, the following feature vectors can be mentioned.
1) Detailed classification feature vector obtained from character pattern (288 dimensions)
Extract a 288-dimensional feature vector from a character pattern using a known technique called multi-dimensional compression (Hai, Kabuyama, Yamamoto "A method of handwritten Kanji recognition" Recognition by IEICE) vol.J68-D no.4, pp773-780, 1985). In this method, first, a direction line element is obtained for each contour point of a character pattern with attributes in four directions, the character pattern is nonlinearly normalized, and then divided into a different number of regions for each direction. Then, the number of pixels of the direction line element is counted for each region, and the count values are arranged in a prescribed order and vectorized. Thereby, a feature vector for detailed classification is obtained.
2) Large classification feature vectors obtained from character patterns (16 dimensions)
Based on the above-described detailed classification feature vector, eigenvalues and eigenvectors are obtained for each category, and 16 axes (eigenvectors) from the larger eigenvalue are obtained. Then, a 16-dimensional vector having a value obtained by projecting the representative vector of the category on each axis is generated. Thereby, a feature vector for large classification is obtained.
[0023]
When these feature vectors are used, when the feature vector in the recognition dictionary corresponding to the character code of one character in the keyword and the feature vector of the character in the search target document are matched, the distance between the same characters By using the threshold value, it can be determined that they are the same character. Further, different characters can be distinguished from each other by using a threshold of the same distance.
c) Character information
・ Character position
For example, the coordinate values of the upper left and lower right vertices of the circumscribed rectangle of the black pixel connection area of the character are used.
・ Size information
This information represents the size of a representative character rectangle included in the line to which the character belongs. For example, the mode value of the height and width of the character rectangle included in the line to which the character belongs is used.
・ Representative value of distance between characters in line
For example, the average value of the distance between the circumscribed rectangles of the characters in the line is used.
-Information indicating the sequence of character rectangles in the line and the order of the character rectangles
For example, in the case of a horizontal writing line, as shown in FIG. 3, 0, 1, 2,. . . Use a serial number such as Then, the gap number on the left side of the rectangle is set as the start number of one character rectangle, and the gap number on the right side is set as the end number to generate character rectangle order information.
[0024]
Of the character rectangles in FIG. 3, rectangles 1 to 9 correspond to basic rectangles, and rectangles 10 to 15 correspond to integrated rectangles obtained by integrating a plurality of basic rectangles. For example, the rectangle 10 is generated by integrating the rectangle 1 and the rectangle 2, and the rectangle 14 is generated by integrating the rectangle 1, the rectangle 2, and the rectangle 3. These rectangles represent a plurality of different cutout results for one row. Also, the order information of these rectangles is as shown in FIG.
d) Line information
·line number
The line number represents a unique number assigned to each line in the page.
・ Row direction
The row direction represents vertical or horizontal.
[0025]
FIG. 5 is a flowchart of the full-text search process based on the output search information. First, the user inputs a keyword desired to be searched (step S11). Next, the search device extracts characters (key characters) constituting the keyword from the search target file storing the search information (step S12), and there is a character string corresponding to the keyword based on the key character. A region is estimated (step S13).
[0026]
Next, the image information in the area is compared with the information in the recognition dictionary to verify whether or not the keyword is included in the area, and a character string that matches the keyword is extracted (step S14). Then, the position information of the extracted character string is output as a search result (step S15), and the process ends.
[0027]
Next, a specific example of the full-text search process will be described with reference to FIGS.
6 and 7 are flowcharts of the key character extraction process performed in step S12 of FIG. In this process, a key character search based on the certainty factor of the recognition candidate is performed. Then, the character code of one character in the input keyword matches the character code of the character in the search target file (including the second and lower recognition candidates), and the certainty factor has a certain threshold value. If it exceeds, the character is extracted as a key character.
[0028]
First, the search device creates an array keycode that holds each character in the keyword (step S21 in FIG. 6), and puts the number of characters in the keyword into a variable keyword_num (step S22). Next, 0 is put into a variable extract_num representing the number of extracted key characters (step S23), and one character as a recognition result is extracted from the search target file (step S24).
[0029]
Next, 1 is put into the variable i representing the rank of the recognition candidates (step S25), 1 is put into the variable j representing the index of the array keycode (step S26), and the character code of the i-th candidate is changed to j of the keycode. The character code of the second character is compared (step S27).
[0030]
If the two character codes match, the confidence of the i-th candidate is compared with a certain threshold value (step S28). If the certainty factor exceeds the threshold value, the character extracted in step S24 is regarded as a key character, and the j-th character code of the keycode and the position of the extracted character are recorded (step S29). Then, extract_num = extract_num + 1 is set (step S30).
[0031]
Next, j = j + 1 is set (step S31 in FIG. 7), and j is compared with keyword_num (step S32). If j is less than keyword_num, the processes in and after step S27 are repeated.
[0032]
If the two character codes do not match in step S27, and if the certainty factor is less than or equal to the threshold value in step S28, it is determined that the character extracted in step S24 is not a key character, and step S31. Perform the following processing.
[0033]
If j exceeds keyword_num in step S32, then i = i + 1 is set (step S33), and i is compared with the number of recognition candidates (step S34). If i is equal to or less than the number of recognition candidates, the processes in and after step S26 are repeated.
[0034]
If i exceeds the number of recognition candidates, it is next checked whether or not the characters in the search target file have ended (step S35). If unprocessed characters remain, the processing from step S24 onward is repeated for the next character. Then, when all characters are processed, the process is terminated.
[0035]
In this process, an evaluation value or a distance value can be used instead of the certainty factor of the recognition candidate. When the evaluation value is used, the evaluation value of the recognition candidate is compared with a certain threshold value in step S28, and if the evaluation value exceeds the threshold value, the character of the search target file is regarded as a key character. If the distance value is used, the distance value of the recognition candidate is compared with a certain threshold value in step S28, and if the distance value falls below the threshold value, the character of the search target file is regarded as a key character.
[0036]
According to the key character extraction process described above, since each character in the search target file is collated with a plurality of recognition candidates and the characters constituting the keyword, the key character in the document image is easily detected. Further, since the key character is extracted based on the certainty factor, the evaluation value, or the distance value of the recognition candidate, the key character extraction accuracy is improved.
[0037]
8 and 9 are flowcharts of the keyword region estimation process performed in step S13 of FIG. In this process, the position of the character in the keyword is obtained from the character information of the extracted key character. Then, an estimated keyword region surrounding the key character is obtained from the position in the keyword and character range information such as the number of characters and the character width of the keyword. For example, in the case of horizontal writing, a keyword region extending from the key character to the left and right is generated, and in the case of vertical writing, a keyword region extending from the key character to the top and bottom is generated.
[0038]
First, the search device puts 1 in a variable i indicating one of the extracted key characters (step S41 in FIG. 8), and puts 0 in a variable kwnum representing the number of extracted keyword regions (step S42). Information on the character code of the i-th key character and the position in the search target file is acquired (step S43).
[0039]
Next, the keyword area in the document image is determined from the character information such as the position and size of the key character (the width or height of the character in the line to which the key character belongs), the character spacing distance, and the position of the key character in the keyword. Estimate (step S44).
[0040]
For example, consider a case where “Fujitsu Laboratories” is input as a keyword and “communication” is extracted as a key character. In this example, as shown in FIG. 10, an xy coordinate system is set in the document image, and the extracted character information of the character “T” is the coordinates (x, y) = (200) of the upper left vertex of the character rectangle. , 500) and the coordinates (250, 550) of the lower right vertex are recorded as positions. Further, it is recorded that a typical character width is 50 dots and a character interval distance is 10 dots.
[0041]
In this case, as the coordinate values of the upper and lower ends of the estimated keyword area, the y coordinate 500 of the upper left vertex and the y coordinate 550 of the lower right vertex are respectively adopted, and the coordinate values of the left end and the right end are calculated by the following equations. .
Left coordinate:
X coordinate of upper left vertex-(character width + distance between characters) x (number of characters to the left of the key character)
= 200- (50 + 10) * 2
= 80
Right coordinate:
X coordinate of the lower right vertex + (character width + character spacing distance) x (number of characters to the right of the key character)
= 250 + (50 + 10) × 3
= 430
Therefore, if the range of the estimated keyword area is expressed using the coordinates of the upper left and lower right vertices, (80,500)-(430,550) is obtained.
[0042]
Next, the search device puts 1 in the variable j indicating one of the already registered keyword areas, puts 0 in the overlap flag variable kwnum2 with the already extracted keyword area (step S45), and the estimated keyword area is j It is checked whether or not it overlaps with the th already registered keyword area (step S46).
[0043]
If the two keyword areas overlap, a rectangular area surrounding the keyword areas is obtained as a new keyword area, and the coordinates of the jth registered keyword area are updated (step S47). As a result, the range of the j-th registered keyword area is replaced with the new keyword area range. Then, kwnum2 = 1 is set.
[0044]
Next, j = j + 1 is set (step S48), and j and kwnum are compared (step S49 in FIG. 9). If j is less than or equal to kwnum, the processing from step S46 is repeated, and if j exceeds kwnum, the value of kwnum2 is checked next.
[0045]
If kwnum2 = 0, an estimated keyword area is newly registered, and kwnum = kwnum + 1 is set (step S51). Next, i = i + 1 is set (step S52), and it is checked whether there is information on the i-th extracted key character (step S53). If there is an i-th extracted key character, the processing from step S43 is repeated, and if the extracted key character information ends, the processing ends.
[0046]
In step S46, if the two keyword regions do not overlap, the process from step S48 is performed as it is. If kwnum2> 0 in step S50, the process from step S52 is performed.
[0047]
According to such a keyword area estimation process, it is possible to accurately estimate the range of the keyword area including the character based on the position of the extracted key character. Even if the first character of a keyword is not extracted as a key character, if another character is extracted as a key character, the corresponding keyword region can be estimated from the position of the character. Furthermore, when a plurality of key characters constitute one keyword area, the keyword area can be correctly extracted.
[0048]
FIG. 11 is a flowchart of the keyword extraction process performed in step S14 of FIG. In this process, for the character rectangle in the range of the estimated keyword area obtained, the character pattern image is matched with the corresponding character in the recognition dictionary, and it is verified whether the character string in the area is really a keyword. .
[0049]
For example, all non-overlapping rectangle combinations in the keyword area are obtained, and for each of the obtained combinations, the image feature obtained from the character pattern in each rectangle matches the image feature of the keyword constituent character obtained from the dictionary. These combinations correspond to different character cutout results. Then, when there is a combination whose degree of matching is equal to or greater than a certain threshold (for example, the sum of distance values is equal to or less than a certain threshold), it is considered that the keyword has been detected.
[0050]
For example, when the character rectangle shown in FIG. 3 is included in the keyword area, the following combinations of rectangles are conceivable.
・ Rectangle 1 to Rectangle 9
・ Rectangle 10 and Rectangle 3 to Rectangle 9
Rectangle 10, rectangle 13, and rectangles 4 to 9
Rectangle 10, rectangle 11, and rectangles 5 to 9
Rectangle 10 to rectangle 12 and rectangle 7 to rectangle 9
Rectangle 1, rectangle 2, rectangle 11, and rectangle 5 to rectangle 9
Rectangle 1, rectangle 2, rectangle 11, rectangle 12, and rectangle 7 to rectangle 9
Rectangle 1 to rectangle 4, rectangle 12, and rectangle 7 to rectangle 9
Rectangle 1, rectangle 13, and rectangle 4 to rectangle 9
Rectangle 1, rectangle 13, rectangle 4, rectangle 12, and rectangle 7 to rectangle 9
Rectangle 14 and rectangle 4 to rectangle 9
Rectangle 14, rectangle 4, rectangle 12, and rectangle 7 to rectangle 9
Rectangle 14, rectangle 15, and rectangle 6 to rectangle 9
Rectangle 10, rectangle 3, rectangle 15, and rectangle 6 to rectangle 9
Rectangle 1, rectangle 13, rectangle 15, and rectangle 6 to rectangle 9
FIG. 11 shows the verification process for one estimated keyword area, but the search device performs the same process for all estimated keyword areas, and outputs only the area where the keyword is detected as a search result. . It is assumed that the rectangles included in each combination in the estimated keyword area are sorted in advance from the closest to the top of the corresponding line.
[0051]
First, the search device pays attention to one of the combinations of rectangles in the keyword area, and selects the first rectangle of the combination as the current rectangle (step S61). Next, 0 is set in the variable num indicating the number of characters in the area matching the characters in the keyword, and 1 is set in the variable k indicating the characters in the keyword (step S62). Call (step S63). In the recursive matching process, each character pattern in the region is matched with the keyword character pattern using the image feature, and the number of matched characters is set as num as a return value.
[0052]
Next, the return value num is compared with the number of characters of the keyword (step S64), and if they do not match, it is checked whether there is another combination in the same keyword area (step S65). If there is another combination, the processing from step S61 onward is repeated for that combination.
[0053]
In step S64, if the num matches the number of keyword characters, the number of areas in which the keyword is detected is incremented (step S66), and the process of step S65 is performed. In step S65, when the verification of all the combinations in the keyword area is completed, the process is terminated.
[0054]
In the recursive matching process in step S63, first, a rectangular character pattern and character information in the keyword area are acquired. The verification methods used in the subsequent processing include the following methods.
a) Method of verifying all rectangular character patterns
For all rectangles in the region, alignment processing using image information and threshold processing are performed to verify whether the target rectangle is a character constituting a keyword.
[0055]
Specifically, all rectangles in the region are matched based on image information, and rectangles with a distance equal to or less than a threshold value are regarded as being matched with keyword characters. Then, it is considered that the keyword has been extracted when all the characters of the keyword match the rectangle in the area. According to this method, since all the rectangles are verified using the image information, the keyword extraction accuracy is improved.
b) Method for verifying only rectangular character patterns with low confidence
Excludes rectangles in the area whose certainty satisfies a predetermined condition, and performs matching processing and threshold processing using image feature information for the remaining rectangles.
[0056]
Specifically, out of the rectangles in the region, the rectangle whose confidence level is higher than the threshold value is considered to be already matched with the keyword character, and the rectangle whose confidence level is equal to or lower than the threshold value is determined by the image information. Perform matching. A rectangle with a distance equal to or less than the threshold value is regarded as being matched with the keyword character, and if all the keyword characters are matched with the rectangle in the region, the keyword is regarded as being extracted. According to this method, the processing time can be reduced without impairing the keyword extraction accuracy.
[0057]
12 and 13 are flowcharts of recursive matching processing using the verification method of a). First, the search device compares the start number of the current rectangle with the end number of the previous rectangle (step S71 in FIG. 12). If they do not match, it is next checked whether the start number of the current rectangle is 0 (step S72). If the start number of the current rectangle is 0, it is next checked whether or not the line number of the current rectangle is the line number of the previous line + 1 (step S73).
[0058]
If the line number of the current rectangle is the line number +1 of the previous line, the image feature (feature vector, etc.) of the character in the current rectangle is compared with the image feature of the kth character of the keyword (step S74). ). If the distance between the image features is equal to or smaller than the threshold value, it is considered that the current rectangular character matches the k-th character of the keyword, and num = num + 1 is set (step S75).
[0059]
Next, k + 1 is compared with the number of characters of the keyword (step S76 in FIG. 13). If k + 1 is equal to or less than the number of keyword characters, k = k + 1 is set (step S77), and the next rectangle after the current rectangle is set as a new current rectangle (step S78). As the next rectangle, a rectangle having the end number of the current rectangle as the start number is selected. Then, the recursive matching process is called to obtain the return value num (step S79), the num is returned to the caller (step S80), and the process ends.
[0060]
In step S71, if the start number of the current rectangle matches the end number of the previous rectangle, the processing from step S74 is performed. If the start number is not 0 in step S72, the line number is not the line number +1 of the previous line in step S73, and if the character of the current rectangle does not match the kth character of the keyword in step S74, The process of step S80 is performed as it is. In step S76, the process of step S80 is also performed when k + 1 exceeds the number of keyword characters.
[0061]
When the verification method of b) is used, the same processing as in FIG. 12 and FIG. 13 is performed on the rectangles between the rectangles with high certainty instead of all the rectangles in the region. In this case, in step S64 of FIG. 11, the number of remaining characters (partial character strings) excluding characters with high certainty among the characters in the keyword is used as the number of keyword characters.
[0062]
Moreover, in the verification method of b), the same process can be performed using an evaluation value or a distance value instead of the certainty factor. In this case, the matching process and the threshold process are performed on the remaining rectangles except for the rectangles in the keyword area whose evaluation value or distance value satisfies a predetermined condition.
[0063]
The pattern extraction apparatus of this embodiment can be configured using, for example, an information processing apparatus (computer) as shown in FIG. 14 includes a CPU (central processing unit) 111, a memory 112, an input device 113, an output device 114, an external storage device 115, a medium driving device 116, a network connection device 117, and an image input device 118. They are connected to each other by a bus 119.
[0064]
The memory 112 includes, for example, a ROM (read only memory), a RAM (random access memory), and the like, and stores programs and data used for processing. The CPU 111 performs necessary processing by executing a program using the memory 112.
[0065]
The input device 113 is, for example, a keyboard, a pointing device, a touch panel, and the like, and is used for inputting instructions and information from the user. The output device 114 is, for example, a display, a printer, a speaker, or the like, and is used for outputting an inquiry to a user and a processing result.
[0066]
The external storage device 115 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, or the like. The information processing apparatus stores the above-described program and data in the external storage device 115, and can use them by loading them into the memory 112 as necessary. The external storage device 115 is also used as a database for storing search target files.
[0067]
The medium driving device 116 drives the portable recording medium 120 and accesses the recorded contents. As the portable recording medium 120, any computer-readable recording medium such as a memory card, a floppy disk, a CD-ROM (compact disk read only memory), an optical disk, or a magneto-optical disk is used. The user can store the above-described program and data in the portable recording medium 120 and load them into the memory 112 for use as necessary.
[0068]
The network connection device 117 is used for connection to an arbitrary communication network such as a local area network (LAN) and performs data conversion accompanying communication. The information processing apparatus can receive the above-described program and data from an external apparatus via the network connection apparatus 117, load them into the memory 112, and use them.
[0069]
The image input device 118 is an imaging device such as a CCD camera or a scanner, and is used for inputting a document image including a color image.
FIG. 15 shows a computer-readable recording medium that can supply a program and data to the information processing apparatus of FIG. Programs and data stored in the portable recording medium 120 or the external database 121 are loaded into the memory 112. Then, the CPU 111 executes the program using the data and performs necessary processing.
[0070]
In the embodiment described above, rectangular coordinates are used as character position information. However, in the present invention, coordinates of other arbitrary shapes can be used. Moreover, you may use shapes other than a rectangle as an estimation keyword area | region.
(Supplementary Note 1) A search device for searching information in a document image using information of a recognition result of the document image,
An input means for inputting a keyword;
Extracting means for extracting any character constituting the keyword from the document image as a key character;
Estimating means for estimating an area corresponding to the keyword in the document image based on the position information of the key character;
An output means for outputting a search result based on the estimated area information;
A search device comprising:
(Supplementary note 2) The search according to supplementary note 1, wherein the extracting means extracts a key character by comparing a plurality of recognition candidates of characters in the document image with an arbitrary character constituting the keyword. apparatus.
(Additional remark 3) The said extraction means extracts the said key character using one of the certainty of the recognition candidate of the character in the said document image, an evaluation value, and a distance value, The additional remark 1 characterized by the above-mentioned. The described search device.
(Additional remark 4) The said estimation means calculates the range of the area | region corresponding to the said keyword using at least one among the size information of the character in the said document image, and the information of a character space | interval, and the positional information on the said key character. The search device according to supplementary note 1, wherein:
(Additional remark 5) It further has a verification means which verifies whether the character string in this estimated area | region is the said keyword based on the character pattern in the said estimated area | region, The said output means contains this keyword The search apparatus according to claim 1, wherein information on a region including a character string verified to be is output as the search result.
(Additional remark 6) The said verification means performs the alignment process and threshold value process which used image feature information for all the character patterns in the said estimated area | region, and whether the object character pattern is a character which comprises a keyword. The search device according to appendix 5, wherein the search device is verified.
(Additional remark 7) The said verification means remove | excludes the character pattern in which one of reliability, an evaluation value, and a distance value satisfy | fills predetermined conditions from the character pattern in the said estimated area | region, and the remaining character pattern 6. The search apparatus according to appendix 5, wherein a matching process and a threshold process using image feature information are performed on the subject to verify whether the target character pattern is a character constituting a keyword.
(Supplementary Note 8) The verification unit performs matching processing and threshold processing using image feature information for each of a plurality of combinations of character patterns corresponding to a plurality of character cutout results, and each combination corresponds to the keyword. 6. The search device according to appendix 5, wherein whether or not to verify is verified.
(Additional remark 9) It further has the production | generation means which produces | generates the information for a search which recognizes the said document image beforehand as a search result file containing a recognition result, and the storage means which stores this information for a search as a search object file, The said extraction means and estimation The search apparatus according to appendix 1, wherein the means performs processing using the search information stored in the storage means.
(Supplementary Note 10) A recording medium on which a program for a computer that searches information in a document image using information of a recognition result of a document image is recorded,
The program is
Extracting from the document image any character that constitutes a keyword as a key character,
Based on the position information of the key characters, an area corresponding to the keyword in the document image is estimated,
Output search results based on estimated area information
A computer-readable recording medium that causes the computer to execute processing.
(Supplementary Note 11) A search method for searching information in a document image using information on a recognition result of the document image,
Specify keywords,
An arbitrary character constituting the keyword is extracted as a key character from the document image,
Based on the position information of the key characters, an area corresponding to the keyword in the document image is estimated,
Present search results based on estimated region information
A search method characterized by that.
[0071]
【The invention's effect】
According to the present invention, a character recognition result is effectively used in a search process for information in an image, and a highly accurate search can be performed. As a result, even for words in document images that could not be searched due to misrecognition in the past, a full-text search using keywords is possible.
[Brief description of the drawings]
FIG. 1 is a principle diagram of a search device according to the present invention.
FIG. 2 is a flowchart of search information generation processing;
FIG. 3 is a diagram illustrating a character rectangle.
FIG. 4 is a diagram illustrating order information of character rectangles.
FIG. 5 is a flowchart of a full text search process.
FIG. 6 is a flowchart (part 1) of key character extraction processing;
FIG. 7 is a flowchart (No. 2) of key character extraction processing;
FIG. 8 is a flowchart (part 1) of keyword region estimation processing;
FIG. 9 is a flowchart (part 2) of the keyword area estimation process;
FIG. 10 is a diagram showing keyword regions.
FIG. 11 is a flowchart of keyword extraction processing.
FIG. 12 is a flowchart (part 1) of recursive matching processing;
FIG. 13 is a flowchart (part 2) of recursive matching processing;
FIG. 14 is a configuration diagram of an information processing apparatus.
FIG. 15 is a diagram illustrating a recording medium.
[Explanation of symbols]
101 Input means
102 Extraction means
103 Estimating means
104 Output means
111 CPU
112 memory
113 Input device
114 output device
115 External storage device
116 Medium drive device
117 Network connection device
118 Image Input Device
119 bus
120 portable recording media
121 database

Claims (4)

A search device for searching for information in a document image using information on a recognition result of the document image,
An input means for inputting a keyword;
An extraction unit that searches the document image for an arbitrary character constituting the keyword, and uses the searched character as a key character to extract position information of the key character ;
Based on the position information of the key character, the character number information indicating the number of characters at the head of the key character among the characters constituting the keyword, the representative size of the character in the document image, and the representative using at least one of character spacing information indicating a character spacing, and calculates the tip position of the region corresponding to the keyword in the document image, at the end side of the key characters of the characters constituting the keyword Using the number of characters and at least one of the character size information and the character spacing information, the end position of the region corresponding to the keyword is calculated, and the range between the obtained tip position and the end position is calculated. An estimation means for estimating the region corresponding to the keyword;
A search apparatus comprising: output means for outputting, as a search result , information on a region including a character string corresponding to the keyword among the estimated regions.   Verification means for verifying whether the character string in the estimated area is the keyword based on the character pattern in the estimated area, and the output means is the keyword The search apparatus according to claim 1, wherein information on a region including the verified character string is output as the search result. A recording medium storing a program for a computer that searches information in a document image using information on a recognition result of the document image,
The program is
From the document image , search for an arbitrary character constituting a keyword, using the searched character as a key character, extract the position information of the key character ,
Based on the position information of the key character, the character number information indicating the number of characters at the head of the key character among the characters constituting the keyword, the representative size of the character in the document image, and the representative using at least one of character spacing information indicating a character spacing, and calculates the tip position of the region corresponding to the keyword in the document image, at the end side of the key characters of the characters constituting the keyword Using the number of characters and at least one of the character size information and the character spacing information, the end position of the region corresponding to the keyword is calculated, and the range between the obtained tip position and the end position is calculated. Estimate the region corresponding to the keyword,
A computer-readable recording medium that causes the computer to execute a process of outputting information on a region including a character string corresponding to the keyword among the estimated regions as a search result. A search method for searching for information in a document image using information on a recognition result of the document image,
Input means, enter a keyword,
The extracting means searches the document image for an arbitrary character constituting the keyword, and uses the searched character as a key character to extract position information of the key character ,
The estimation means uses the position information of the key character as a reference, and the character size indicating the number of characters on the front side of the key character among the characters constituting the keyword and the representative size of the character in the document image Using at least one of the information and character spacing information indicating a representative character spacing , the leading edge position of the region corresponding to the keyword in the document image is calculated, and the key character among the characters constituting the keyword The end position of the region corresponding to the keyword is calculated using the number of characters on the end side and at least one of the character size information and the character spacing information, and between the obtained tip position and the end position. Is assumed to be the region corresponding to the keyword,
Search method output means, among the estimated area, the information of a region including a character string corresponding to the keyword, and outputs as a search result.

Images