JP7396568B2 - Form layout analysis device, its analysis program, and its analysis method - Google Patents

Description

The present invention relates to a form layout analysis device for analyzing the layout of a form, an analysis program thereof, and an analysis method thereof.

Conventionally, a method is known in which a form is captured as image data and the layout of the form is analyzed through image processing. For example, Patent Document 1 discloses a document editing and output device that analyzes the document structure in a document image using a template that defines the layout of the document. Additionally, Patent Document 2 discloses that by allowing the data type of a field to be set semi-automatically using a learning function, more detailed OCR constraints can be easily imposed, thereby improving the accuracy of character recognition. An information processing device is disclosed. Specifically, the format information and constraints corresponding to the input form image are read from the format model storage unit, and the values entered in the form image in the field specified by this format information are within the range of the field constraints. Characters are recognized.

Japanese Patent Application Publication No. 11-219442 JP 2017-10069 Publication

However, the above-mentioned analysis methods of Patent Document 1 and Patent Document 2 assume that the template and format of the form are registered in the system in advance, and therefore cannot be applied to unknown forms that are not registered in the system.

Therefore, an object of the present invention is to enable layout analysis of unknown forms that are not registered in the system.

In order to solve this problem, a first invention provides a form layout analysis device that includes an item extraction section and a layout analysis section and analyzes the layout of a form. The item extraction unit uses a deep learning-based object detection algorithm to extract item regions that include item names printed in type on the form, and individually classifies the item regions included in the form image with attribute classification. Extract. The layout analysis unit analyzes the layout of the form image based on the position of the item area on the form image and its attributes.

Here, in the first invention, a handwritten region extracting section may be further provided that individually extracts handwritten character regions including character strings written in handwritten characters on the form from the document image. In this case, it is preferable that the layout analysis section assigns one of the attributes classified by the item extraction section to each of the handwritten character regions extracted by the handwritten region extraction section. Further, the layout analysis unit associates the item area with a handwritten character area located in the vicinity thereof according to a preset correspondence rule, and then associates the item area with the handwritten character area located in the vicinity thereof. Attributes of the area may also be assigned.

In the first invention, the item extraction unit inputs the form image into one neural network as an object detection algorithm, and extracts the item area and classifies its attributes at the same time using a regression approach. It is preferable to do so. In this case, the item extraction unit refers to a learning model constructed by supervised learning using teacher data that is a pair of an item image including an item name expressed in print and an attribute of the item image. It is also possible to extract item areas and classify their attributes.

In the first invention, the item extraction unit may output the classification accuracy of the attribute. In this case, it is preferable to further provide a filter processing unit that removes attributes whose classification accuracy is smaller than a predetermined threshold value as noise.

In the first invention, when a plurality of analysis results are obtained regarding the layout of the form image, the layout analysis section may present the plurality of analysis results to the user as layout candidates.

A second invention provides a form layout analysis program that causes a computer to execute processing having the following steps to analyze the layout of a form. In the first step, the item regions containing the item names printed in type on the form are extracted, and an object detection algorithm based on deep learning is used to extract the item regions contained in the form image individually with attribute classification. Extract to. In the second step, the layout of the form image is analyzed based on the position of the item area on the form image and its attributes.

Here, in the second invention, a third step may be further provided in which handwritten character areas including character strings written on the form in handwritten characters are individually extracted from the form image. In this case, the second step preferably assigns one of the attributes classified in the first step to each of the handwritten character regions extracted in the third step. In addition, in the second step, the item area and the handwritten character area located in the vicinity thereof are associated with each other according to a preset correspondence rule, and then the handwritten character area is associated with the handwritten character area. Attributes of item areas may also be assigned.

In the second invention, the first step is to input the form image into one neural network as an object detection algorithm, and use a regression approach to extract item areas and classify their attributes. It is preferable to do so. In this case, the first step is to refer to a learning model constructed by supervised learning using training data that is a pair of item images containing printed item names and attributes of this item image. , extraction of item areas and classification of their attributes may be performed.

In the second invention, the first step may include a step of outputting the classification accuracy of the attribute. In this case, it is preferable to further provide a fourth step of removing attributes whose classification accuracy is smaller than a predetermined threshold value as noise.

In the second invention, the second step may include, when a plurality of analysis results are obtained regarding the layout of the form image, a step of presenting the plurality of analysis results as layout candidates to the user.

A third invention provides a form layout analysis method that includes the following steps and analyzes the layout of a form. In the first step, the item regions containing the item names printed in type on the form are extracted, and an object detection algorithm based on deep learning is used to extract the item regions contained in the form image individually with attribute classification. Extract to. In the second step, the layout of the form image is analyzed based on the position of the item area on the form image and its attributes.

Here, in the third invention, a third step of individually extracting handwritten character regions including character strings written in handwritten characters on the form from the form image may be further provided. In this case, the second step preferably assigns one of the attributes classified in the first step to each of the handwritten character regions extracted in the third step. In addition, in the second step, the item area and the handwritten character area located in the vicinity thereof are associated with each other according to a preset correspondence rule, and then the handwritten character area is associated with the handwritten character area. Attributes of item areas may also be assigned.

In the third invention, the first step is to input the form image into one neural network as an object detection algorithm, and use a regression approach to extract item areas and classify their attributes. It is preferable to do so. In this case, the first step is to refer to a learning model constructed by supervised learning using training data that is a pair of item images containing printed item names and attributes of this item image. , extraction of item areas and classification of their attributes may be performed.

In the third invention, the first step may include a step of outputting the classification accuracy of the attribute. In this case, it is preferable to further provide a fourth step of removing attributes whose classification accuracy is smaller than a predetermined threshold value as noise.

In the third invention, the second step may include, when a plurality of analysis results are obtained regarding the layout of the form image, a step of presenting the plurality of analysis results as layout candidates to the user.

According to the present invention, item areas included in a form image and their attributes are acquired using an object detection algorithm based on deep learning. From this information, it is possible to specify what information is written at which position in the form image. This makes it possible to perform layout analysis even for unknown forms that are not registered in the system.

Block diagram of a form layout analysis device according to this embodiment Diagram of object detection algorithm YOLO network configuration diagram Layout analysis flowchart Diagram showing an example of a form image Diagram showing item areas extracted from a form image Diagram showing the handwritten character area extracted from the form image Diagram showing layout analysis results of form images Explanatory diagram of layout candidates by extracting multiple attributes An explanatory diagram of layout candidates based on the proximity of multiple item areas

FIG. 1 is a block diagram of a form layout analysis device according to this embodiment. This form layout analysis device 1 analyzes the layout of a form on which handwritten character strings are written, such as an application form or a contract, and specifies what is written where on the form. The form to be analyzed is an unknown one, that is, a form whose layout is not registered in the system, and is performed, for example, as preprocessing prior to optical character recognition (OCR) of a handwritten form. The form layout analysis device 1 mainly includes an item extraction section 2, a handwritten area extraction section 3, a filter processing section 4, a layout analysis section 5, a learning processing section 6, a learning model 7, and a correspondence rule table 8. It is composed of

The item extraction unit 2 extracts item areas included in the form image individually with attribute classification, using item areas including item names printed in type on the form as extraction targets. For example, if there are image areas such as "name" and "address" in a form image, each image area is extracted as an item area, and the image areas such as "name" and "address" are extracted for each item area. Attributes are added. Extraction of item areas is performed using an object detection algorithm based on deep learning, and the extraction of item areas and classification of their attributes are performed with reference to the learning model 7 constructed based on this algorithm. Furthermore, for the classified attributes, the classification accuracy is also calculated and output.

FIG. 2 is an explanatory diagram of the object detection algorithm. As shown in FIG. 5A, in the conventional detection method used for face detection, etc., processing for input is divided into three stages: area search, feature extraction, and machine learning. That is, a region search is performed, features are extracted according to the object to be detected, and an appropriate machine learning method is selected. In this detection method, object detection is realized using three algorithms. As for feature quantities, they are basically designed specifically for the detection target, so only specific targets can be detected. Therefore, in order to eliminate such constraints, an object detection algorithm using deep learning as shown in FIGS. 2(b) and 3(c) has been proposed. As shown in FIG. 6(b), features such as R-CNN (Regions with Convolutional Neural Network) automatically perform feature extraction using deep learning. This allows flexible classification of various objects just by designing the network. However, since region search still remains as a separate process, there are some methods that include region search in deep learning, such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector), as shown in the same figure (c). This is the method. In this method, by inputting an input (form image) into one neural network, extraction of item areas and classification of their attributes are performed at the same time. The first feature of this method is that it is a regression-like approach. Regression is an approach that directly predicts numerical values from trends in data. Rather than determining an area and then classifying what it is, the coordinates and size of an object are directly predicted. Second, processing can be completed in one network. It can be said that it is an "end-to-end" process in the sense that after inputting the data, deep learning goes all the way to the end (output results). The present embodiment is characterized in that items on a form are extracted using an object detection algorithm based on deep learning, and in particular, the method shown in FIG.

For example, YOLO processing is generally as follows. First, the input image is divided into S*S regions. Next, the class probabilities of objects within each region are derived. Then, parameters (x, y, height, width) and confidence of B bounding boxes (hyperparameters) are calculated. The bounding box is a circumscribed rectangle of the object region, and the reliability is the degree of agreement between the predicted and correct bounding boxes. For object detection, the product of the object's class probability and the reliability of each bounding box is used. FIG. 3 is a diagram of the YOLO network configuration. In YOLO, a form image is input to a CNN (Convolutional Neural Network) layer, and the result is output after passing through multiple stages of fully connected layers. The output includes an image region divided into S*S pieces, five parameters of a bounding box (BB) including reliability (classification accuracy), and the number of classes (item attributes).

The handwritten area extracting unit 3 individually extracts handwritten character areas including character strings written in handwritten characters on a form from the form image. Various methods have been proposed for distinguishing between handwritten characters and printed characters, and any method can be used. For example, a method of analyzing character images in real space may be used. Specifically, we have developed methods for extracting character strings by taking horizontal and vertical histograms of characters, and extracting handwritten character strings by evaluating the linearity of the baseline, and This is a method of extracting handwritten character strings by evaluating linearity and character symmetry. At this time, variations in the size of individual handwritten characters constituting the character string, the degree of proximity between handwritten characters, etc. may be considered. Instead of analysis in real space, a method of analyzing character images in frequency space may be used. Furthermore, the handwritten region extracting unit 3 may use a model that identifies "type/print", "handwritten", "ruled line", "stamp", and "background" on a pixel basis. In this case, a method such as Semantic Segmentation may be used as a classifier.

Among the multiple attributes extracted by the item extraction unit 2, the filter processing unit 4 considers the attributes whose reliability (classification accuracy) is smaller than a predetermined threshold as noise and removes them. Information regarding the item area filtered by the filter processing section 4 is output to the layout analysis section 5.

The layout analysis unit 5 analyzes the layout of the form image based on the position of the item area on the form image and its attributes, and specifies where information related to which attribute is written. Specifically, one of the attributes classified by the item extraction unit 2 is assigned to each of the handwritten character areas extracted by the handwriting area extraction unit 3. Basically, when a certain item area and a certain handwritten character area are close to each other on a form image, that is, the distance between the two is less than or equal to a predetermined threshold, a correspondence is established between the two, and this handwritten character area The attributes of this item area are assigned to. For example, if a handwritten character area exists near an item area that has the attribute "name", the attribute "name" is assigned to this handwritten character area. Further, specific correspondence rules between item areas and handwritten character areas are set and defined in advance in the correspondence rule table 8. In addition to basic rules such as matching the handwritten character area when there is a handwritten character area near the right side of the item area, or matching the two when there is a handwritten character area near the bottom of the item area, It also defines how to handle handwritten character areas that exist in tables.

The learning processing unit 6 constructs a learning model 7 by supervised learning using teacher data that is a pair of an item image (partial image) containing an item name expressed in print and an attribute of this item image. The learning model 7 referred to by the item extraction unit 2 can be reconstructed after the fact as the amount of teacher data increases.

The flow of document layout analysis will be described in detail below with reference to FIGS. 5 to 8. FIG. 4 is a flowchart of layout analysis performed by the form layout analysis device 1. Note that this form layout analysis device 1 can also be equivalently realized by installing a computer program (form layout analysis program) that causes the computer to function and operate as blocks 2 to 6 in a computer.

First, in step 1, a form image to be analyzed is input. FIG. 5 is a diagram showing a "transfer request form" as an example of a form image. In this form, item names such as "name," "affiliation," "amount," and "bank name" are printed in type on the form. Further, in the blank space located near each item name, items corresponding to the item name are handwritten.

Next, in step 2, the item extracting section 2 extracts item areas existing in the form image individually with attributes. As shown in FIG. 6, for the print string "furigana", a rectangular area including this is extracted as an item area a1, and its attribute "phonetic" and classification probability are assigned. Regarding the print string "department", a rectangular area including this is extracted as the item area a2, and its attribute "department" and classification probability are assigned. Regarding the print string "name", a rectangular area including this is extracted as an item area a3, and its attribute "name" and classification probability are assigned. Regarding the print string "amount", a rectangular area including this is extracted as an item area a4, and its attribute "amount" and classification probability are assigned. Regarding the print string "bank", a rectangular area including this is extracted as the item area a5, and its attribute "bank" and classification probability are assigned. Regarding the print string "branch name", a rectangular area including this is extracted as an item area a6, and its attribute "branch" and classification probability are assigned. Regarding the print string "deposit type", a rectangular area including this is extracted as an item area a7, and its attribute "account type" and classification probability are given. Regarding the print string "account number", a rectangular area including this is extracted as an item area a8, and its attribute "account number" and classification probability are assigned.

Next, in step 3, handwritten character regions existing in the form image are individually extracted by the handwritten region extraction unit 4. As shown in FIG. 7, for the handwritten character string "Tokyo Ichiro", a rectangular area including this is extracted as the handwritten character area b1. Regarding the handwritten character string "Intellectual Property Department", a rectangular area including this is extracted as a handwritten character area b2. Regarding the handwritten character string "Patent Ichiro", a rectangular area including this is extracted as a handwritten character area b3. Regarding the handwritten character string "6,500", a rectangular area including this is extracted as the handwritten character area b4. Regarding the handwritten character string "Sumitomo Mitsui," a rectangular area including this is extracted as a handwritten character area b5. Regarding the handwritten character string "Ginza", a rectangular area including this is extracted as a handwritten character area b6. Regarding the handwritten character string "normal", a rectangular area containing this is extracted as a handwritten character area b7. Regarding the handwritten character string "19620123", a rectangular area including this is extracted as the handwritten character area b8.

Next, in step 4, the filter processing unit 4 filters the attributes extracted by the item extraction unit 2, and removes attributes that are considered to be noise.

Next, in step 5, the layout analysis section 5 analyzes the layout of the form image. As shown in FIG. 8, since the handwritten character area b1 is located near the right of the item area a1, the attribute "phonetic" of the item area a1 is assigned. Since the handwritten character area b2 is located near the right of the item area a2, the attribute "department" of the item area a2 is assigned. Since the handwritten character area b3 is located near the right of the item area a3, the attribute "name" of the item area a3 is assigned. Since the handwritten character area b4 is located near the right of the item area a4, the attribute "amount" of the item area a4 is assigned. Since the handwritten character area b5 is located near the bottom of the item area a5, the attribute "bank" of the item area a5 is assigned. Since the handwritten character area b6 is located near the bottom of the item area a6, the attribute "branch" of the item area a6 is assigned. Since the handwritten character area b7 is located near the bottom of the item area a7, the attribute "account type" of the item area a7 is assigned. Since the handwritten character area b8 is located near the bottom of the item area a8, the attribute "account number" of the item area a8 is assigned.

Finally, in step 6, the analysis result of the layout of the form image as shown in FIG. 8 is output, and thus the series of processes ends. Note that in the above processing, Step 2 and Step 3 are independent of each other, so their execution order may be reversed, or they may be executed simultaneously.

Note that if a plurality of analysis results are obtained by the layout analysis in step 5, these analysis results may be presented to the user as layout candidates in order to leave the suitability of the results to the user's judgment. The following two cases can be considered as such cases. The first case, as shown in FIG. 9, is a case where a plurality of attributes 1 and 2 are assigned to one item area a on the form image. In this case, both Attribute 1 and Attribute 2 are considered as attributes of the handwritten character area b located near this item area a, so both Attribute 1 and Attribute 2 are considered as attributes of the handwritten character area b. will be presented. The second case is a case where a plurality of item areas a1 and a2 are close to a certain handwritten character area b, as shown in FIG. In this case, since both attribute 3 of item area a1 and attribute 4 of item area a2 are considered as attributes of handwritten character area b, both attributes 3 and 4 can be considered by the user as attribute candidates for handwritten character area b. Presented.

As described above, according to the present embodiment, item regions included in a form image and their attributes are acquired using an object detection algorithm based on deep learning. From this information, it is possible to specify what information is written at which position in the form image. This makes it possible to perform layout analysis even for unknown forms that are not registered in the system. In particular, as an object detection algorithm using deep learning, it is possible to use a method such as YOLO or SSD that extracts item areas in a form image and classifies its attributes using a single neural network. It becomes possible to detect the target at high speed.

Further, according to the present embodiment, handwritten character areas including character strings written in handwritten characters on a form are individually extracted from the form image, and each of the handwritten character areas is classified by the item extraction unit 2. Assign one of the specified attributes. This makes it possible to perform layout analysis even on forms with handwritten character strings.

1 Form layout analysis device 2 Item extraction unit 3 Handwritten area extraction unit 4 Filter processing unit 5 Layout analysis unit 6 Learning processing unit 7 Learning model 8 Correspondence rule table

Claims (8)

In a form layout analysis device that analyzes the layout of a form,
an item extraction unit that individually extracts item areas included in a form image with attribute classification, with an item area including an item name printed in type on the form as an extraction target;
a layout analysis unit that analyzes the layout of the form image based on the position of the item area on the form image and its attributes;
has
The item extraction unit extracts the item by referring to a learning model constructed by supervised learning using teacher data that is a pair of an item image including an item name expressed in print and an attribute of the item image. Extract regions and classify their attributes
A form layout analysis device characterized by: further comprising a handwritten region extraction unit that individually extracts handwritten character regions including character strings written in handwritten characters on the form from the form image,
The layout analysis unit assigns one of the attributes classified by the item extraction unit to each of the handwritten character areas extracted by the handwriting area extraction unit. Form layout analysis device. The layout analysis unit associates the item area with the handwritten character area located in the vicinity thereof according to a preset correspondence rule, and then associates the item area with the handwritten character area located in the vicinity thereof. 3. The form layout analysis device according to claim 2, wherein attributes of the item area are assigned. 4. The item extraction unit extracts the item area and classifies its attributes at the same time by inputting the form image into one neural network. A form layout analysis device described in The item extraction unit outputs the classification accuracy of the attribute,
5. The form layout analysis device according to claim 1, further comprising a filter processing unit that removes the attributes whose classification accuracy is smaller than a predetermined threshold value as noise. According to any one of claims 1 to 3, the layout analysis unit presents the plurality of analysis results as layout candidates to the user when a plurality of analysis results are obtained regarding the layout of the form image. A form layout analysis device. In a form layout analysis program that analyzes the layout of a form,
a first step of individually extracting item areas included in the form image with attribute classification , with item areas including item names printed in type on the form as extraction targets;
a second step of analyzing the layout of the form image based on the position of the item area on the form image and its attributes;
has
The first step refers to a learning model constructed by supervised learning using training data that is a pair of an item image including an item name expressed in print and an attribute of the item image. Extract item areas and classify their attributes
A form layout analysis program that causes a computer to perform processing. In a form layout analysis method that analyzes the layout of a form,
a first step of individually extracting item areas included in the form image with attribute classification , with item areas including item names printed in type on the form as extraction targets;
a second step of analyzing the layout of the form image based on the position of the item area on the form image and its attributes;
has
The first step refers to a learning model constructed by supervised learning using training data that is a pair of an item image including an item name expressed in print and an attribute of the item image. Extract item areas and classify their attributes
A form layout analysis method characterized by the following.

Images