JP6118646B2 - Form processing device, form processing method, form processing program - Google Patents

Description

  The present invention relates to an image processing technique in an OCR (Optical Character Reader) or the like.

  When a character is read using OCR, generally, steps of binarization, character line extraction, character segmentation, character identification, and language collation are performed. The outline of each processing is as follows. Binarization is processing for generating a binary image in which characters are black and others are white from an input color image or grayscale image. Character line extraction is a process for specifying a line area in which characters are described in an image. Character segmentation is a process of specifying individual character areas in a character line area. Character identification is a process of performing a character recognition process on an image of each character area and obtaining a character recognition result (character code) and a certainty factor. Language collation is a process of collating a character recognition result with a previously registered word or the like and correcting an error in the character recognition result. For example, when the country name is to be recognized, the character recognition result is compared with the dictionary in which the word of the country name is registered, and the most matching word is regarded as the recognition result.

  In addition to character identification, errors also occur in character segmentation. For example, “∝” may be erroneously cut out like “oc”. Such a cut-out error also occurs due to blurring or crushing of characters. In order to avoid such cutout errors, for example, it is conceivable that character recognition is performed on all possible cutout patterns, and the character recognition result that most closely matches in language collation is used as the final result.

  In an OCR for a form, ruled lines and characters are printed on paper as a part of the form in advance (referred to as pre-printing), and the characters described on the line may be recognized. In general, written characters to be read by the OCR are often written in black or blue. The pre-printed portion is often printed in a different color from the written characters in order to distinguish them from the written characters.

  As a method of recognizing a character line from a form in which pre-printing and written characters are mixed, it is conceivable to drop out pre-printed (not recognized as characters) and recognize only written characters. In this case, if the characters to be recognized and the pre-print are described in the same color, there is a problem that language collation cannot be performed by dropping them together.

  The following Patent Document 1 discloses a technique for recognizing both preprinted characters and written characters as a technique for solving the above-described problems. Specifically, binary images extracted in red, green, blue, and black are combined to synthesize an image, character recognition is performed on the combined image, and handwritten characters and pre-printed characters are recognized. ing.

JP 2009-265751 A

  When recognizing characters written on a pre-printed form, the characters may be overlapped on the pre-printed portion. In this case, in the technique described in Patent Document 1, a character cannot be identified because the preprinted portion and the character portion overlap each other even on the composite image. Also, depending on the type of form, the color of pre-printing may not always be determined, so that the pre-printed part and the character part may not be properly separated depending on the technique described in Patent Document 1.

  The present invention has been made in view of the problems as described above, and is a form processing technology that can accurately recognize written characters even when the pre-printed portion and written character portion of the form overlap. It aims at realizing.

  The form processing apparatus according to the present invention recognizes characters for each color from a form image, generates a plurality of candidate character strings by changing the order of recognized characters, and obtains candidate character strings and character string patterns in a notation dictionary. The candidate character string is evaluated by comparing.

  According to the form processing apparatus according to the present invention, even if the pre-printed portion of the form and the written characters overlap, the written characters can be recognized with high accuracy.

2 is a functional block diagram of a form processing apparatus 100 according to Embodiment 1. FIG. It is a figure which shows an example of the character recognition result in case a pre print character and the description character have overlapped. It is a figure which shows the example which isolate | separated the pre-printed character and the description character by extracting a color from a color form image. FIG. 5 is a diagram illustrating an operation flow of the form processing apparatus 100. It is a flowchart explaining the detail of step S406. It is a figure which illustrates the candidate character string produced | generated in step S406. It is a figure explaining the detail of step S407-S408. It is a figure which shows the example in which the output part 180 displays the evaluation result by the character string evaluation part 160 on a screen in a GUI format. It is a figure which illustrates the some binarized image from which the grade of a color dropout differs.

<Embodiment 1>
FIG. 1 is a functional block diagram of a form processing apparatus 100 according to the first embodiment of the present invention. The form recognition apparatus 100 is an apparatus for processing a form image, and includes an image input unit 110, a color extraction unit 120, a character recognition unit 130, a character string generation unit 140, a notation dictionary reading unit 151, a notation dictionary 152, and a character string evaluation unit. 160, a storage device 170, and an output unit 180.

  The image input unit 110 is configured using, for example, a device such as a scanner, and takes a form as image data. The color extraction unit 120 extracts colors in the captured form image. The character recognition unit 130 recognizes a character for each extracted color. The character string generation unit 140 generates a character string that is a candidate for the final recognition result from the character identification result for each color by the process described later with reference to FIGS. The notation dictionary reading unit reads the character string pattern stored in the notation dictionary 152. The character string evaluation unit 160 performs language collation on the candidate character string generated by the character string generation unit 140 and the character string pattern in the notation dictionary 152, and scores and evaluates the collation result. The output unit 180 outputs the evaluation result by the character string evaluation unit 160.

  The notation dictionary 152 is a dictionary database that is referred to when the form processing apparatus 100 processes a form. Specifically, the notation dictionary 152 is a character recognition dictionary that is referred to when the form processing apparatus 100 performs character recognition, a knowledge dictionary that is referred to when language verification is performed, and a layout that is referred to when specifying a form layout. Store information, etc. The knowledge dictionary has items such as date, amount, address, name, and account number. Each dictionary holds a character string pattern exemplified in FIG. The storage device 170 stores form image data to be processed by the form processing apparatus 100.

  The character recognition unit 130 performs binarization of the form image, character line extraction, character segmentation, and character recognition. In character line extraction, for example, a rectangular region where characters are expected to be described can be connected, and a rectangle including the whole can be regarded as a character line.

  Each of the above functional units can be realized by using hardware such as a circuit device that realizes the function, or realized by executing a software that implements the function by an arithmetic unit such as a CPU (Central Processing Unit). You can also

  The notation dictionary 152 and the storage device 170 can be realized by storing data in a storage device such as a hard disk device. The output format of the output unit 180 may be, for example, a data format that is output via an arbitrary communication line, or may be a format that is output as a GUI (Graphical User Interface) on the screen.

  FIG. 2 is a diagram illustrating an example of a character recognition result when the pre-printed characters and the written characters overlap. In the color form image 201, the written character 203 overlaps the preprinted character 202 due to printing misalignment. The binarized image 204 is an image obtained by binarizing the color form image 201. An area 205 is an area where the preprint character 202 and the written character 203 overlap. A character line 206 indicates a result of extracting a character line part in the binarized image 204. The recognition result 207 is a result of character identification performed by cutting out characters from the character line 206. A portion where the pre-printed character 202 and the written character 203 do not overlap can be identified, but the overlapping portion is identified as an erroneous character with no identification result.

  As illustrated in FIG. 2, when the pre-printed character and the written character overlap, even if character recognition is performed on the binarized image, the character recognition result of the overlapping portion is obtained. It is difficult. Therefore, in the first embodiment, the preprinted portion and the written character portion are separated by the method described below.

  FIG. 3 is a diagram illustrating an example in which pre-printed characters and written characters are separated by extracting colors from a color form image. The binarized image 301 is a binarized image in which other colors are dropped out while leaving the described character color. A character line 302 is a result of extracting a character line from the binarized image 301. The character string 303 is a character string obtained as a result of character extraction from the character line 302 and character identification. The binarized image 304 is a binarized image in which other colors are dropped out while leaving the pre-print color. A character line 305 is a result of extracting a character line from the binarized image 304. The character string 306 is a character string obtained as a result of character extraction from the character line 305 and character identification. Since the identification value of each character is associated with the coordinate value in the form image, the space portion between the characters may be omitted.

  As illustrated in FIG. 3, by using a binarized image generated for each color, it is possible to recognize characters by distinguishing between written characters and pre-printed characters. However, since the respective character recognition results are independent of each other, language collation cannot be performed on a character string in which these are integrated. Therefore, in the first embodiment, a candidate character string is generated by further combining preprinted characters and written characters. Details of the candidate character string will be described later with reference to FIG.

  FIG. 4 is a diagram for explaining the operation flow of the form processing apparatus 100. Hereinafter, each step of FIG. 4 will be described.

(FIG. 4: Steps S401 to S403)
The image input unit 110 acquires a color form image (S401). The color extraction unit 120 extracts the pre-print color and the description character color from the color form image (S402). At this time, the approximate colors may be regarded as the same color. The following steps S404 to S405 are repeated for each color extracted in step S402.

(FIG. 4: Step S404)
The character recognition unit 130 generates a binarized image by leaving any one color extracted in step S402 and dropping out the other colors. The character recognition unit 130 further performs layout analysis on the generated binarized image and extracts a character line. For example, it is possible to extract ruled lines and dotted lines in a form image, extract a table portion based on the extracted ruled lines and dotted lines, and extract character lines from inside and outside the table. About the layout of the whole form image, you may implement by the same method, for example, you may recognize using the layout information in the description dictionary 152. FIG.

(FIG. 4: Step S405)
The character recognition unit 130 performs character recognition in order to identify each character in the character line extracted in step S404. Specifically, a character string pattern is generated by cutting out characters from a character line, and each character is identified by comparing the generated character pattern with the character pattern held in the character recognition dictionary in the notation dictionary 152. . The character recognition unit 130 stores the coordinate values and identification results of the identified character patterns in the form image in the storage device 170.

(FIG. 4: Step S406)
The character string generation unit 140 generates a candidate character string that matches the character string pattern held in the knowledge dictionary in the notation dictionary 152 using the character recognized in step S405. Specifically, a plurality of character string patterns are generated as candidate character strings by switching the order of pre-printed characters and written characters. Details of this step will be described later with reference to FIG.

(FIG. 4: Steps S407 to S408)
The character string evaluation unit 160 performs language collation by comparing the candidate character string generated in step S406 with the character string pattern held in the knowledge dictionary in the notation dictionary 152 (S407). The character string evaluation unit 160 assigns a score to the candidate character string based on the result of language collation (S408).

(FIG. 4: Step S409)
The output unit 180 outputs the evaluation result by the character string evaluation unit 160. A screen example when the evaluation result is output as a GUI will be described with reference to FIG.

  FIG. 5 is a flowchart for explaining details of step S406. Hereinafter, each step shown in FIG. 5 will be described.

(FIG. 5: Steps S501 to S502)
The character string generation unit 140 sets the written character and the pre-printed character recognized in step S405 as the initial value of the candidate character string (S501). The character string generation unit 140 checks whether or not the written characters and the pre-printed characters overlap in line units based on the coordinate values associated with each character (S502). If the lines overlap, the process proceeds to step S503. If the lines do not overlap, the candidate character string generated in step S501 is used as it is.

(FIG. 5: Step S502: Supplement)
The character string generation unit 140 may provide a certain allowable range when determining whether the written character and the pre-printed character overlap in line units. For example, if the height of the portion where the written character and the pre-printed character overlap is within half of the character height, it can be considered that they are written in the same line. As a result, even if the written characters are slightly shifted in the height direction, it can be processed as being written in the same line as the pre-printed characters.

(FIG. 5: Step S503)
The character string generation unit 140 sorts the written characters and the pre-printed characters by the left end coordinate based on the coordinate value of each character obtained by the character recognition in step S405. Thus, for example, the character string described in the character line 302 of FIG.

(FIG. 5: Step S504)
Based on the coordinate value of each character, the character string generation unit 140 checks whether the written character and the pre-printed character overlap in character units. If they overlap in character units, the process proceeds to step S505. If they do not overlap, the character string generated in step S503 is set as a candidate character string.

(FIG. 5: Step S505)
The character string generation unit 140 generates candidate character strings that are candidates for the final result of character recognition by switching the character order of written characters and the character order of pre-printed characters. For example, a candidate character string can be generated by changing the character order based on the following hypothesis.

(FIG. 5: Step S505: Example of method for changing the character order)
(Method Example 1) Assuming that the written character and the pre-printed character are overlapped in character units by shifting the written character to the left, the written character is shifted to the right by X characters:
(Method Example 2) Assuming that the written character and the preprinted character are overlapped in character units by shifting the written character to the right, the written character is shifted to the left by X characters:
(Method 3) Assuming that the written characters and the pre-printed characters are overlapped in units of characters due to the large shift of the written characters to the right, first set all the written characters in order from the left, followed by the pre-printed characters Generate a string:
(Method 4) Assuming that the written characters and the pre-printed characters are overlapped in units of characters because the written characters are greatly shifted to the left, first, all the pre-printed characters are set in order from the left, followed by the written characters Generate a string.

(FIG. 5: Step S505: Supplement)
The character recognition unit 130 can also determine that a space character is included in the written character when the character spacing of the written character is greater than or equal to the height of the character line. The same applies to pre-printed characters. The character string generation unit 140 can also generate a candidate character string by inserting a pre-printed character in a space portion in the written character or inserting a written character in the space portion in the pre-printed character. When each character is described in a plurality of colors, the characters to be inserted can be combined for each color.

  FIG. 6 is a diagram illustrating the candidate character string generated in step S406. The candidate character string table 601 is a list of candidate character strings. The written characters are represented by dark characters, and the preprinted characters are represented by light characters. Candidate character strings 602 and 603 are generated in step S501. Candidate character strings 607 and 608 are respectively generated by (Method Example 3) (Method Example 4).

  FIG. 7 is a diagram illustrating the details of steps S407 to S408. For the sake of explanation, the character string pattern held in the notation dictionary 152 is illustrated. Collation table 701 shows the result of collating notation dictionary 152 and candidate character strings in steps S407 to S408. First, the configuration of the notation dictionary 152 will be described, and then the collation table 701 will be described.

  The notation dictionary 152 can be described in, for example, an RTN (Recursive Transition Network) format. When collating the candidate character string with the notation dictionary 152, for example, the candidate character string is replaced with a network format that takes into account the ambiguity of character extraction called the candidate character network and the ambiguity of the character identification result. Can be verified. The format and collation method of the notation dictionary 152 are not limited to these, as long as the character string pattern held in the notation dictionary 152 can be compared with the candidate character string.

  Date notation patterns 1521 and 1522 are examples in which date notation is expressed in the RTN format. YYYY represents the year, MM represents the month, and DD represents the day. A parenthesis indicates that it can be omitted, and indicates that there is either one before or after “|”. For example, (year |. | −−) indicates that the character string pattern is matched if any of “year”, “.”, And “−” exists. Furthermore, since the character string pattern is enclosed in parentheses, the collation can be omitted. The money amount notation patterns 1523 and 1524 are examples in which the money amount notation is expressed in the RTN format. “N” represents a number, and “^” represents the repetition of the number N. These notations are examples, and the character string patterns held in the notation dictionary 152 are not limited to these.

  The collation table 701 shows the result of the character string evaluation unit 160 collating the candidate character strings 602 to 608 shown in FIG. 6 with the notation dictionary 152 shown in FIG. The candidate character string 602 matches “YYMMDD” of the date notation pattern 1521 and also matches “NNNNNN” of the money amount notation pattern 1524. The candidate character string 606 matches the “YY year MM month DD day” of the date notation pattern 1521. The candidate character strings 603 to 605 and 607 to 608 do not match any of the notation patterns.

The character string evaluation unit 160 assigns a score to the candidate character string that matches the notation pattern using, for example, the following calculation formula.

  In the collation table 701, since the candidate character string 606 has the highest score, the candidate character string 606 can be used as the final result of character string recognition. Therefore, the character recognition result is regarded as a character string of a date notation pattern. The calculation formula of the score is not limited to the above, and other calculation formulas can also be used.

  FIG. 8 is a diagram illustrating an example in which the output unit 180 displays the evaluation result by the character string evaluation unit 160 in a GUI format. The input image display unit 801 displays a color form image input to the form processing apparatus 100. The target area 802 is an area where character recognition is performed. The character line recognition result column 803 displays the result of recognizing the character line for each color from the target area 802. The collation table column 804 displays the collation table 701 described with reference to FIG. A thick line frame in the collation table column 804 indicates a candidate character string that is a final result of character recognition. The final recognition result 805 displays the final result. The user can also correct the final recognition result 805. In this case, the initial value displayed in the correction input field is the final recognition result 805 (that is, the evaluation character string having the highest evaluation result).

<Embodiment 1: Summary>
As described above, the form processing apparatus 100 according to the first embodiment generates candidate character strings by recognizing characters for each color from the form image and switching the character order, and collates the candidate character strings with the notation dictionary 152. To evaluate whether the candidate character string is correct. Thereby, even if the pre-printed character and the written character overlap, the character string can be recognized with high accuracy.

  Further, the form processing apparatus 100 according to the first embodiment recognizes a space character included in a character line and generates a candidate character string by inserting a character of another color into the space portion. As a result, it is possible to accurately perform character recognition on a pre-printed form intended to describe characters in the space portion.

<Embodiment 2>
In the first embodiment, it has been described that characters are recognized for each color and the character order is changed. When recognizing characters for each color, unrecognized colors are dropped out. At this time, if the written characters and pre-printing are uneven in color, a part of the characters may be lost due to the color dropout. Therefore, in the second embodiment of the present invention, a method of generating a plurality of binarized images having different degrees of color dropout and comparing them will be described. Since the configuration of the form processing apparatus 100 is the same as that of the first embodiment, the operation relating to the shading unevenness will be mainly described below.

  FIG. 9 is a diagram illustrating a plurality of binarized images with different degrees of color dropout. In step S403, the character recognizing unit 130 generates a binarized image by leaving a color value within a predetermined range centered on the color value extracted in step S402 and dropping out other color values. Further, a plurality of binarized images are acquired by changing the threshold value of the color value to be dropped out. As the threshold value of the color to be dropped out, for example, a color value in the RGB color space may be used, or a color value in the HSV color space in which the RGB value is changed to hue, saturation, luminance, or brightness may be used. The color space may be used. Here, it is assumed that binarized images 901 to 906 shown in FIG. 9 have been generated.

  The binarized image 901 is generated by setting a small threshold (reducing the remaining range) around the written character color. The binarized image 903 is generated by setting a large threshold (increase the remaining range) around the written character color. The binarized image 902 is set with a threshold value that is approximately between 901 and 903. The binarized image 904 is generated by setting a small threshold with the pre-printed character color as the center. The binarized image 906 is generated by setting a larger threshold with the pre-printed character color as the center. The binarized image 905 is set with a threshold value that is approximately between 904 and 906.

  The character recognition unit 130 recognizes characters in the character line by comparing a plurality of binarized images generated as shown in FIG. For example, when there are uneven shading in the written characters and some of the characters are missing, it is possible to recognize the characters with little missing by the binarized images 903 and 906 generated using a larger threshold. On the other hand, when the written characters are blurred and part of the characters are crushed, the characters can be recognized with less crushed characters by the binarized images 901 and 904 generated using a smaller threshold. In any case, for example, a character recognition result having the highest likelihood may be adopted.

  In step S504, when it is confirmed whether or not the characters overlap each other, a plurality of binarized images generated from the same character line are not subject to confirmation. This is because these plurality of binarized images have the same character coordinates, and thus naturally overlap.

<Embodiment 2: Summary>
As described above, the form processing apparatus 100 according to the second embodiment generates the plurality of binarized images by changing the range in which the color dropout is performed, and compares the character recognition results with respect to these images to the maximum likelihood. Specific character recognition results are identified. As a result, even if the written characters have shading unevenness or bleeding, character recognition can be performed with high accuracy.

<Embodiment 3>
In Embodiments 1 and 2, there may be a case where a preprinted character and a description character overlap and a portion that does not overlap in the color form image. It is considered that it is sufficient to use a conventional character recognition method for a portion where both do not overlap. Therefore, for example, conventional character recognition may be performed once on an input color form image, and the method described in the first embodiment may be performed only on a portion that cannot be recognized. This eliminates the need to check the overlap of characters on the entire form, leading to a reduction in processing time.

  In the first and second embodiments, the output unit 180 may output only one final recognition result 805, or may output a plurality of recognition results in order from the highest score, for example. In this case, the number of recognition results to be output or the score range may be determined in advance or may be specified by the user.

  The present invention is not limited to the embodiments described above, and includes various modifications. The above embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to the one having all the configurations described. A part of the configuration of one embodiment can be replaced with the configuration of another embodiment. The configuration of another embodiment can be added to the configuration of a certain embodiment. Further, with respect to a part of the configuration of each embodiment, another configuration can be added, deleted, or replaced.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized in hardware by designing a part or all of them, for example, with an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

  100: Form recognition device 110: Image input unit 120: Color extraction unit 130: Character recognition unit 140: Character string generation unit 151: Notation dictionary reading unit 152: Notation dictionary 160: Character string evaluation unit 170: storage device, 180: output unit.

Claims (11)

An apparatus for processing a form image,
An image input unit for receiving the form image;
A color extraction unit for extracting colors included in the form image;
A character recognition unit that recognizes characters described in the form image for each color extracted by the color extraction unit;
A character string generation unit that generates a plurality of candidate character strings by connecting the characters recognized by the character recognition unit for each color;
A notation dictionary reading unit that reads the character string pattern from a notation dictionary that stores a plurality of character string patterns including a character string pattern described in the form image;
A character string evaluator that evaluates the degree of matching between each candidate character string and the character string pattern in the notation dictionary by comparing each candidate character string and the character string pattern in the notation dictionary;
An output unit for outputting the evaluation result by the character string evaluation unit and the candidate character string;
A form processing apparatus comprising: The character string generation unit generates the candidate character string for each color by connecting the characters recognized by the character recognition unit for each color extracted by the color extraction unit,
The character string evaluation unit
When the character strings for each color generated by the character recognition unit overlap in character units in the form image, the plurality of candidate character strings are connected by shifting the positions of the characters in the character strings. Produces
When the character strings for each color generated by the character recognition unit do not overlap in character units in the form image, the character strings for each color generated by the character recognition unit are used as the candidate character strings. The form processing apparatus according to claim 1, wherein When the distance between characters recognized for each color is equal to or greater than a predetermined value, the character recognition unit recognizes that a space character is described between the characters,
The said character string production | generation part produces | generates the said candidate character string by inserting the character of another said color with respect to the part which the said character recognition part recognized as a space character about a certain said color. 2. The form processing apparatus according to 2. The character recognition unit
By dividing the form image into a part having a color value within a predetermined range centered on the color extracted by the color extraction unit and another part, a partial area in which characters are described in the form image is obtained. Extract and
Further, by performing the division while changing the predetermined range, a plurality of the partial areas in which the same character is described are extracted, and the result of recognizing the character using each of the partial areas is compared with the most. The form processing apparatus according to claim 1, wherein a plausible recognition result is specified. The character string generation unit regards a plurality of characters recognized by the character recognition unit within a predetermined range in the height direction of the characters as characters described in the same line and described in the same line. The form processing apparatus according to claim 2, wherein the candidate character string is generated for each color extracted by the color extraction unit by concatenating characters regarded as being processed. The form processing apparatus according to claim 1, wherein the output unit outputs the evaluation result only when the matching degree evaluated by the character string evaluation unit is within a predetermined range. The form processing apparatus according to claim 1, wherein the output unit outputs the character string pattern evaluated by the character string evaluation unit together with the evaluation result. The form processing apparatus according to claim 7, wherein the output unit distinguishes and outputs the character string pattern evaluated by the character string evaluation unit for each color extracted by the color extraction unit. The output unit is configured as a display unit that displays the evaluation result and the candidate character string on a screen,
The display unit has a correction input field for correcting the previous evaluation result,
The form processing apparatus according to claim 7, wherein the initial value displayed in the correction input field is the candidate character string having the highest evaluation result. A method for processing a form image,
An image input step for receiving the form image;
A color extraction step of extracting colors included in the form image;
A character recognition step for recognizing characters described in the form image for each color extracted in the color extraction step;
A character string generation step of generating a plurality of candidate character strings by concatenating the characters recognized for each color in the character recognition step;
A notation dictionary reading step of reading out the character string pattern from a notation dictionary storing a plurality of character string patterns including the character string pattern described in the form image;
A character string evaluation step for evaluating the degree of matching between each candidate character string and the character string pattern in the notation dictionary by comparing each candidate character string and the character string pattern in the notation dictionary;
An output step of outputting the evaluation result in the character string evaluation step and the candidate character string;
A form processing method characterized by comprising:   A form processing program for causing a computer to execute the form processing method according to claim 10.

Images