JP4881722B2 - Form identification device and form identification program - Google Patents

Description

  The present invention relates to a form identification technique for visually displaying the type of form identified in a form identification process.

In general, when performing automatic identification of a form, the type of form is identified using the feature amount in the form. For example, character codes, character lines, ruled lines, frames, etc. described in the form are extracted, and the feature type is compared with the feature quantity of the form registered in advance as a sample to identify the type of form . For example, the technique disclosed in Patent Literature 1 registers feature data of a form as a dictionary for each form, and creates a ruled line binary image from an input image when a form image to be identified is input, This is a method of performing matching with a dictionary registered in advance. In the dictionary, the positions of black pixels obtained from ruled lines and the like are registered as a set of two-dimensional coordinates. In the technique disclosed in Patent Literature 2, point coordinates are added to the features of the input form to be identified (hereinafter referred to as input points), and the features of the registered form registered in advance for creating the dictionary are point coordinates. And a distance between the input point and the dictionary point is calculated. If the distance is within a certain distance criterion, it is determined that a dictionary point associated with the input point exists. In this method, the degree of matching is obtained based on the associated score, input score, and dictionary score. In this method, as input points or dictionary points, the end points and intersections of ruled lines, and the center coordinates of the smallest area (cell) surrounded by the ruled lines are used as feature points.
As described above, there are various identification methods for the forms to be identified. The problem here is how to present the result of form identification on the display device. It is common to display the name of the form type. However, when there are various forms to be handled, it is difficult to determine whether the identification result is correct only by the name.

Japanese Patent Laid-Open No. 10-222587 JP 2002-324236 A

  In the conventional form identification processing, there is a problem that it is difficult to distinguish whether the identification result is correct only by presenting the name of the form type obtained as a result of identification.

  The present invention visually presents a form identification result when handling various types of forms, and even when there are deviations in the form image or when similar forms are mixed in the visual display. It is an object of the present invention to provide a form identification device that highlights and displays differences and a program thereof. It is another object of the present invention to provide a form identification device that can display a sample form image registered in a dictionary or protect personal information described in an input form image.

  In order to achieve the above object, the present invention includes an image capturing unit, a processing unit, a storage unit, and a display unit. The image capturing unit captures a form to acquire image data of the form, and the processing unit When identifying the form image type based on the dictionary stored in the storage unit and displaying the form identification result and the program for displaying the form identification result on the display unit and displaying the form identification result on the display unit In addition, the input form image and the sample form image of the form type having the most matched characteristic obtained by the result of the form identification process are displayed in a superimposed manner.

  In addition, a form identification device and a program for detecting and detecting a shift between the input form image and the sample form image and absorbing the shift using the form feature point mapping are provided. To do.

  Furthermore, the present invention provides a form identification apparatus and program for performing processing such as highlighting a portion having a difference or displaying a portion where personal information can be written with a mask.

  According to the present invention, the form identification result can be displayed by superimposing the input image and the sample image in accordance with the identified form type. Furthermore, even when there is a deviation between the form images, it is possible to perform display processing such as absorbing and displaying the position deviation by using the form feature point mapping, and further highlighting the difference. The efficiency of form distribution processing can be expected.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the hardware configuration assumed in this embodiment will be described. FIG. 1 is a hardware configuration diagram illustrating an embodiment of a form reading apparatus. In the high-speed form reading device, a paper document is converted into electronic data by an image pickup device 0101 which is an image pickup unit, which is stored in an external storage device 0105 and a memory 0106 which are storage units, and is a central processing unit (CPU). Reading is performed by the central processing unit 0107. The form identification / display program according to the present embodiment is stored in the external storage device 0105 or the memory 0106, or is introduced into the form reading device via the communication device 0109 and stored in these storage units depending on circumstances.

  In this form identification / display program, the central processing unit 0107 performs form identification processing and identification result display control for the captured electronic data image. When specifying the form type, the form type is determined by referring to the form dictionary stored in the external storage device 0105 or the memory 0106 as a storage unit. These processes can be operated by a human operator through the operation terminal device 0102, and the processing results are displayed through the display terminal device 0103. Information such as processing results is stored in the external storage device 0105 or transmitted as data to the external device through the communication device 0109 as necessary. The above devices are connected by an internal bus 0108. Reflecting the form identification result, the entered slips are distributed and collected by the sorter device 0104 in a defined box. In other words, except for the image capturing device 0101 and the sorter device 0104, the image capturing device 0101 and the sorter device 0104 can be configured by a computer system such as a normal personal computer (PC). Although a normal PC may have a display control chip separately from the CPU, it goes without saying that in this embodiment, the graphic board may be configured separately to share the display control. Regardless of the configuration, the configuration for executing the form identification and display program may be referred to as a processing unit.

  FIG. 2 is a functional block diagram of the form reading process in the form reading apparatus described above, that is, the process from form identification to distribution. First, an input paper slip is picked up by an image pickup unit 0201 including the image pickup device 0101 to be an electronic (data) image. This electronic image is normally temporarily stored in a storage unit such as the external storage device 0105 or the memory 0106, and subsequent form identification processing is performed based on the captured electronic image. Subsequently, the form identification unit 0202 performs form identification. When form identification is performed, a form identification feature dictionary 0205 in which features for identifying a form are stored in a dictionary is used. The identified form type is controlled to be visually displayed on the display terminal device 0103 by the identification result display unit 0203. As an image displayed by this display control, an image stored in the sample form image dictionary 0206 or an image calculated from the form identification feature dictionary 0205 is used. Finally, according to the confirmed form identification result, according to the distribution definition defined in the form distribution dictionary 0207, the form that is the identification target is conveyed to the corresponding stack of the sorter unit 0204 including the sorter device 0104.

  Here, the form identification unit 0202 and the identification result display unit 0203 are configured by software processing by a program executed by the central processing unit 0107, which is a processing unit, as described above. Next, the outline of the form identification processing unit 0202 and the form identification feature dictionary 0205 will be described first.

  FIG. 3 is a flowchart showing a form identifying process in the form identifying unit 0202 of the present embodiment, which is executed by the central processing unit 0107 as a processing unit. The electronic image of the input form image to be discriminated is identified through this process. First, an electronic image of a form to be discriminated is acquired from an image capturing device 0101 which is an image input unit, which is directly or directly stored in a storage unit and used as an input image (0301). Next, in order to set the feature value extraction range from the input image, paper area reference extraction for detecting the four corners of the form paper is performed (0302). The paper size is calculated from the coordinates of the four corners of the detected form paper, and the paper size of the form to be determined is determined (0303). The feature quantity of the input image is extracted from the paper size determined in the above process from the input image, and the feature quantity is firstly collated using the form identification feature dictionary 0205 created from the form learning sample image (0304).

  The feature amount here refers to a portion where the luminance of the image appears remarkably in the input image extracted from the form to be discriminated, for example, as in the case of the dictionary creation described above. Feature points are extracted and used in a grid pattern. Then, the feature quantity extracted from the input image is compared with the dictionary created from the sample image. If the distance calculation value of both feature quantities is within a certain distance reference value, the input image and the sample image match and match. Yes (YES), and the process ends (0305). As the distance calculation method, Euclidean distance or dynamic entry mask distance is used. When the case corresponds to the case where the input form is rotating, the form is rotated 180 degrees or 90 degrees at a time, and the feature amount is generated, and the distance is calculated. When the calculated distance between the feature amount and the dictionary exceeds the distance reference value, a description area reference extraction for detecting an area (referred to as a description area) described in the form sheet from the input image is performed. (0306). The feature quantity of the input image is newly extracted from the range of the detected description area, the feature quantity is secondarily collated, and it is determined whether or not the input image and the sample image match (0307), and the form is identified. End the process.

  FIG. 4 is a flowchart showing the process of display processing for presenting the form identification result to the display terminal device 0103 in the present embodiment. In the flow of the identification result display unit 0203 executed by the central processing unit 0107 as the processing unit, input information includes an input form image and a form identification result (0401). The input form image is the previous electronic image. The form identification result includes at least the recognized form type, its identification likelihood, the rotation direction, and the degree of deviation. First, if the degree of deviation included in the form identification result exceeds a threshold or the identification likelihood exceeds the threshold (0402), it is determined that there is a deviation between the input form image and the sample form image. Perform calculations to properly overlay the images. First, feature points are extracted from the input form image and the sample form image (0403). Next, feature points are collated (0404), and parameter calculation (calculation of shift amount, rotation amount, expansion / contraction amount) for optimally overlaying the input form image and the sample form image is performed.

  Based on the above result, a sample form image used for overlay display is created (0405). For example, if there is rotation, an image when the sample form image is rotated is calculated. If it is first determined that there is no deviation, the above feature point calculation process can be omitted. Algorithms that take correspondence between feature points and obtain a linear transformation parameter that optimally approximates both images are collectively referred to as feature point mapping. The calculation processing of steps 0403 to 0404, which is a process of deviation calculation, will be described later. Next, after the sample image calculation (0405) is completed, display image calculation of how to display the input form image and the sample form image is performed (0406). Then, the calculated image is displayed on the display of the display terminal device 0103 (0407), and the processing of the identification result display unit 0203 is finished.

  FIG. 5 is a diagram showing a specific example of the feature points extracted and calculated in step 403 in the above flow. As the feature points, for example, a black ruled line of a form is extracted, and the center point of a frame composed of four sets of upper, lower, left, and right black ruled lines is used as a feature point, or a local feature of an image such as a Harris feature is used. There is a method of calculating a concentration variation direction and using the maximum value as a feature point. The method shown in this figure is an example of contour analysis. Here, assuming that a binary image has been input (upper part of FIG. 5), the outline of the connected component is extracted therefrom and approximated by a straight line (middle part of FIG. 5). This is a feature point (lower part of FIG. 5). In the feature point extraction 0403 and the feature point matching 0404 described above, for example, feature point extraction as shown in FIG. 5 is performed on the input form image and the sample form image, and then correspondence between the feature points is taken. Thus, it is possible to calculate exactly how much deviation or rotation exists between the two images.

  RANSAC (Random Sample Consensus) is a typical method for extracting only correct data from a mixture of correct feature points and incorrect feature points. In this case, a necessary minimum number of data are randomly selected from all data, and then a constraint condition parameter (linear transformation) is calculated. Then, the validity of the value is evaluated from the remaining data, and this is performed a sufficient number of times to obtain the value obtained with the maximum evaluation and the data supporting it. This allows both correct parameters and correct data to be estimated simultaneously. In the calculation of the RANSAC algorithm, three points are required. However, when considering the types of linear transformation such as rotation, shift, and expansion / contraction (and XY equal magnification), it is actually sufficient to select the correspondence between two points. By utilizing this property, it is possible to reduce the amount of calculation for form identification. This is called form feature point mapping.

  That is, as shown in FIG. 6, assuming that the point {x1, x2} is a feature point obtained from the sample form and the point {p1, p2} is a feature point obtained from the input form, the point x1 is used as a reference point. , The coordinate of the point x2 is expanded / contracted and rotated so that the vector x2-x1 coincides with the vector p2-p1, and then the operation of moving the point x1 to the point p1 is performed. FIG. 7 shows a pseudo program code for form feature point mapping with a reduced amount of calculation. The linear transformation in the pseudo code is expressed by the following formula. That is, first, the linear transformation

And model. At this time,

Then, the conversion amounts of expansion / contraction, rotation, and movement can be obtained as follows. That is, the amount of expansion and contraction is

It is expressed. The amount of rotation is

It is expressed. The amount of movement is

It is guided.

  FIG. 8 is a specific example of the display in which the input form image and the sample form image are superimposed according to the first embodiment. Here, a coarse mesh image (referred to as a coarse image) included in the form identification feature dictionary 0205 is used as a dictionary, and the two are superimposed. In this case, the sample image dictionary 0206 shown in FIG. 2 becomes unnecessary, and the dictionary capacity can be reduced. The advantage of the coarse image display is that it is possible to avoid the leakage of personal information entered in the input form or the like at the site where the form identification is operated, and the amount of calculation for image processing is reduced.

  Next, FIG. 9 shows a display example in which the input form image and the sample form image are overlaid as another embodiment of the display process, and further, the different differences are highlighted by color display or gray display 90.

  The difference is calculated as follows. It is assumed that the process of identifying the form type of the input form image and the form feature point mapping between the sample form image representing the form type have been completed at the difference portion calculation stage. Here, the form type is identified by the processes 0302 to 0307 shown in FIG. Form feature point mapping is performed by the processes 0402 to 0404 shown in FIG.

  By the above-described form feature point mapping, the amount of rotation, shift, and expansion / contraction to be performed when the two images are overlaid is determined. Therefore, the input form image is converted using the parameter, and the feature vector is calculated using the feature points arranged in a two-dimensional grid. Let I be a vector obtained from the input form image subjected to linear transformation, and M be a vector obtained from the sample form image. Since each vector is extracted using two-dimensional grid-like feature points, these vectors can be regarded as an image that roughly approximates a form. Therefore, a vector corresponding to the input form image is represented by I (x, y), and a vector corresponding to the sample form image is represented by M (x, y). At this time, the difference index D (x, y) representing the difference between the two images is

It can be calculated with That is, the part where D (x, y) is large is a place where there is some difference between the two form images. However, if this difference index D value is used as it is, there is a problem that it is easily affected by local fluctuations, and therefore, the difference likelihood L (x, y) is used as a scale representing a stable difference. The difference likelihood L is an average value in the partial area of the difference index, and is calculated according to the following equation.

Rx and Ry in this formula represent the size of partial averaging. Next, the designated N points (x1, y1)... (XN, yN) are selected in order from the highest difference likelihood. A part corresponding to these points is determined as a difference part on the image, and the color is emphasized and displayed at the part.

  The advantage of the difference highlighting is that, in the case where there are similar form groups, it is possible to support the judgment of success or failure of the result of form identification and to realize quick processing. Needless to say, the calculation of these differences can be realized by the program processing in the central processing unit 0107 constituted by the CPU described above. In addition, a technique for highlighting the difference portion 90 that is a specified area on the screen by color display or gray display when displaying an image can be realized by a display control program executed by the central processing unit 0107 or the like. .

  Next, FIG. 10 shows a display example in which a specific information (description content) written on a form image (referred to as a specific area) is displayed with a mask 100 as another embodiment of the display process. There are two ways to apply the mask. One is a method in which a sample form image is an unfilled form image, a portion different from the sample form image and a black pixel having a threshold value or more is determined as a description area, and the description area is masked and displayed. . The other is a method in which the sample form image is an entry form, and the area (entry area) considered to be handwritten entry is masked and displayed by analysis of the size of the connected component and frequency analysis of the outline component. . The advantage of the mask display is that it is possible to avoid the leakage of personal information entered in the input form or the like at the form identification operation site.

  Incidentally, as means for discriminating whether the character written on the entry form is a handwritten character or a type character originally printed on the form, Japanese Patent Application Laid-Open No. 2003-308483 filed earlier by the present applicant. The method disclosed in the publication “Character Type Determination Program, Character Identification Program, and Character Identification Device” can be used. It goes without saying that the specific area detection for mask display can be realized by the program processing in the central processing unit 0107 constituted by the CPU described above. In addition, a technique for mask-displaying a specific area on the screen when displaying an image can be realized by a display control program executed by the central processing unit 0107 or the like.

  Furthermore, the display methods described above are not only performed independently, but also can be used in combination. For example, a combination of blur display and difference emphasis display is combined, and mask display and difference emphasis display are combined.

  FIG. 11 shows a display example in which the input form and the sample form image are superimposed and displayed as they are in the simplest case as another embodiment of the display process.

  FIG. 12 is an example of a screen configuration when the form identification result is shown to the user when the above-described display method is used. 1201 displays the number of the unknown form as an identification target, and 1202 lists the names of the form types listed as candidates as a result of the form identification. It is also possible to display the reduced images side by side at the same time the form types are arranged. In 1203, buttons for selecting which of them to be displayed on the image are arranged. In this case, since the form D which is the first candidate is selected by the user, the sample form image of the form D and the input form image are superimposed and displayed on 1204. With this display form, the user can select the correct form type while comparing the form images. If a correct form type is not found among the identification candidates, an input box 1205 is provided so that the user can input the form type name. If the user determines that the selected form type is correct, the user presses a decision button 1206 to confirm the form identification result. The processing related to these display screens is also executed by the central processing unit 0107.

The hardware block diagram of the high-speed form reading apparatus used for a 1st Example. The block block diagram of the form reading process in a 1st Example. The processing flowchart of the form identification process in a 1st Example. The processing flowchart of the form identification result display process in a 1st Example. The figure which shows the example of feature point extraction from the form image in a 1st Example. The figure which imaged the correspondence between the feature points in a 1st Example. The figure which showed the pseudo code of the form feature point mapping in a 1st Example. The figure which shows the example 1 of a superimposition display of the input form image and sample form image in a 1st Example. The figure which shows the example 2 of superimposition display of the input form image and sample form image in another Example. The figure which shows the example 3 of an overlay display of the input form image and sample form image in another Example. The figure which shows the example 4 of a superimposed display of the input form image and sample form image in another Example. The figure which shows the example of a confirmation screen structure of a form identification result.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 0101 ... Image pick-up device, 0102 ... Operation terminal device, 0103 ... Display terminal device, 0104 ... Sorter device, 0105 ... External storage device, 0106 ... Memory, 0107 ... Central processing unit, 0108 ... Internal bus, 0109 ... Communication device, 0201 Image capturing unit, 0202 ... Form identifying unit, 0203 ... Identification result display unit, 0204 ... Sorter unit, 0205 ... Form identifying feature dictionary, 0206 ... Sample form image dictionary, 0207 ... Form distribution dictionary, 0301 ... Form image input unit , 0302 ... Paper area reference extraction unit, 0303 ... Size determination part, 0304 ... Primary collation part, 0305 ... Threshold judgment part, 0306 ... Description area reference extraction part, 0307 ... Secondary collation part, 0401 ... Input data, 0402 ... Deviation Determination unit, 0403 ... feature point extraction unit, 0404 ... feature point collation unit, 0405 ... sample image calculation unit, 04 6 ... display image calculating unit, 0407 ... the image display unit.

Claims (6)

A form identification device that reads and identifies forms,
An image capturing unit that reads the form to acquire an input form image, a storage unit that stores a form identification feature dictionary describing feature amounts of a plurality of sample form images, a processing unit that processes the input form image, and a display And
The processor is
Based on the dictionary stored in the storage unit, identify the type of the input form image,
When displaying the result of the form identification on the display unit, the input form image, the sample form image having the most consistent feature amount as a result of the form identification , the input form image and the sample form image When the degree of deviation exceeds a predetermined threshold, the feature point extraction of the input form image and the sample form image is performed, and the input form image and the A form identification device that calculates conversion parameters for superimposing sample form images, converts the sample form images according to the conversion parameters, and controls the display so as to display them superimposed. The form identification device according to claim 1,
The processor is
A form identifying apparatus that controls to display a coarse image in a superimposed manner when the input form image and the sample form image are displayed in a superimposed manner. The form identification device according to claim 1,
The processor is
A form identification device that controls to display a difference portion between the input form image and the sample form image in an emphasized manner when displaying the input form image and the sample form image. The form identification device according to claim 1,
The processor is
A form identification device that controls to display a specific area of the input form image with a mask when the input form image and the sample form image are displayed in a superimposed manner. A form identification program for use in a form identification apparatus including a storage unit for storing a form identification feature dictionary describing feature quantities of a plurality of sample form images, and a processing unit for processing an input form image obtained by reading an input form,
The processing unit is
An identification step for identifying the input form image and which of the plurality of sample form images matches;
When displaying the identification result obtained by the identification step , the degree of deviation between the input form image and the sample form image is determined in advance for the sample form image having the feature quantity that most closely matches the input form image. If the threshold value exceeds the threshold value, the feature points of the input form image and the sample form image are extracted, and the input form image and the sample form image are superimposed based on the extracted matching result of the feature points. A form identification program for controlling to execute a display control step for controlling to display a superimposed parameter after calculating a conversion parameter and converting the sample form image with the conversion parameter . The form identification program according to claim 5,
In the display control step, the processing unit is
When the input form image and the sample form image are overlaid, either a rough image is overlaid, a difference portion is highlighted, or a specific area is masked or displayed. A form identification program that controls to execute combinations.

Images