JP2020046860A - Form reading apparatus - Google Patents

Abstract

To provide a form reading apparatus capable of extracting desired information from filled-out forms.SOLUTION: One embodiment of the present invention relates to a form reading apparatus that comprises an extraction unit for extracting predetermined information from filled-out forms using a learned model.SELECTED DRAWING: Figure 3

Description

  The present invention relates generally to financial technology, and more particularly to a form reading system.

  Currently, when using financial services in banks, customers execute many financial transactions such as deposits, withdrawals, transfers, foreign exchange transactions, etc. on an electronic basis by operating Internet banking etc. with the development of FinTech. It is possible. On the other hand, there is also a financial transaction in which a customer still fills in and seals a paper document or form. For example, an account transfer request form for regularly debiting a utility bill, a credit card usage fee, a purchase price for a product or service from a customer account is a typical example of a paper-based form-based financial transaction.

  The debit request form filled out and stamped by the customer is sent to the bank where the customer's account is located, and it is mainly performed by hand that confirmation of the entry contents is complete, verification of the stamped seal, input to the in-line system, etc. I have. For example, in megabanks, a huge number of account transfer request documents are processed by bank staff every day, and there is a strong demand for automation and efficiency of office work related to processing the request documents. For example, for such a purpose, a form recognition method for extracting a pair of an item name and an item value at high speed from a form image in which many character strings are described is known.

JP 2012-208589 A

  However, according to a conventional form recognition method based on a character recognition model, information in a specific type of form can be read, but a form recognition method capable of reading desired information from many types of forms is not known. In the actual paperwork, a unique fund transfer request form with a different format and layout is prepared for each service such as utility bill, credit card use price, purchase price etc.For example, the types of fund transfer requests handled by megabanks Number over thousands of types. For this reason, an extraction model that can collectively read a plurality of types of forms with different formats and layouts is desired.

  As described above, the fund transfer request form is prepared in different formats and layouts for each client, and is necessary for executing the bank transfer such as client ID information (client code, etc.), customer account information, and seal imprint information. There are various places where information is described. In addition, the debit request form contains various types of information, such as check marks and circles, for selecting relevant items, along with handwritten letters and numbers. Is also required. For this reason, it is difficult to cope with the conventional character recognition model.

  In view of the above-described problems, an object of the present invention is to provide a form reading apparatus capable of extracting desired information from a filled form.

  In order to solve the above-described problem, one embodiment of the present invention relates to a form reading device including an extraction unit that extracts predetermined information from a filled-out form using a learned model.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to extract desired information based on various types of entry contents from forms of various formats and layouts.

1 is a schematic diagram showing a form reading system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a hardware configuration of the form reading device according to one embodiment of the present invention. 1 is a block diagram illustrating a functional configuration of a form reading device according to an embodiment of the present invention. FIG. 1 is a schematic diagram illustrating a neural network structure according to an embodiment of the present invention. 5 is a flowchart illustrating a model learning process according to an embodiment of the present invention. It is a flowchart which shows the form reading process by the learned model by one Example of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the following embodiment, a form reading apparatus that extracts various types of information from filled forms is disclosed. Briefly describing an embodiment described later, the form reading apparatus uses a learned model such as a neural network which inputs a plurality of types of filled forms and outputs predetermined information based on the contents of the forms, Various information is extracted from the input form. For example, when the form reader obtains an account transfer request as an input, the form reader uses the learned model to transfer the account transfer information such as the entrustor code, account information (such as account number), and imprint information of the account transfer request. Extract predetermined information necessary for execution.

  Although the following embodiment is mainly described with respect to a fund transfer request form, the form according to the present invention is not limited to this, and may be applied to any other form filled in / stamped on a paper medium, for example. Good. Further, the following embodiment focuses on information related to financial services such as the contents of an account transfer request form as information to be extracted, but information to be extracted according to the present invention is not limited to this. Any other entry information that can be read from a form, such as personal information such as a name, address, and date of birth in a public service or the like, may be used.

  First, a form reading system according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a schematic diagram showing a form reading system according to one embodiment of the present invention.

  As shown in FIG. 1, the form reading system 10 includes a user terminal 50, a learning device 60, a storage 70, and a form reading device 100.

  The user terminal 50 is typically a terminal for providing various instructions and / or data to the form reading apparatus 100, and is, for example, any information processing apparatus having a communication function such as a personal computer. Is also good.

  The learning device 60 learns a learning target model using teacher data composed of a pair of a form and information to be extracted. For example, when the model to be learned is configured as a neural network, the learning device 60 inputs image information indicating a form for teachers to the neural network, and calculates an error between output information and information to be extracted for teachers. For example, the parameters of the neural network are updated according to back propagation. When the predetermined termination condition is satisfied, the learning device 60 stores the neural network based on the updated parameters in the storage 70 as a learned model. For example, the learned model may be learned for each piece of information to be extracted, or may be learned for all or a part of the information to be extracted.

  For example, when extracting a consignor code from multiple types of completed fund transfer request forms, the case where the name of the consignor is pre-printed on the fund transfer request form, the customer writes the consignor on the fund transfer request form The form of the debit request form, such as the case specified by the customer or the case where the customer specifies by hand writing (for example, writing a check mark, encircling with a circle, etc.) the column of the specific consignor from the selected consignor. There are various variations according to. The layout area for specifying the entrustor also varies depending on the layout of the fund transfer request form. In order to obtain a neural network for extracting entrustor information, entrustor code, etc. from such a variety of fund transfer request forms, the learning device 60 uses a plurality of filled-in account transfers provided from the user terminal 50 or the like. Appropriate pre-processing (rearrangement, normalization, data compression, ruled line extraction, etc. for each consignor code) is performed on teacher data composed of pairs of request forms and consignor codes, and the pre-processed teacher Using the data, a neural network to be learned is learned according to any appropriate learning algorithm (for example, Resnet, Xception, SENet, etc.).

  In addition, when extracting the location and account information of the account information from the completed multiple types of account transfer request forms, there are also various variations according to the format of the account transfer request form. The layout area in which the location and the account information are entered also varies depending on the layout of the account transfer request form. Furthermore, account numbers and the like are typically in a form to be entered in squares separated by ruled lines, and a model capable of appropriately distinguishing ruled lines from written contents is required. In order to obtain a neural network for extracting the account information entry location and account information from such various types of account transfer requests, the learning device 60 uses a plurality of entered account transfers provided from the user terminal 50 or the like. Appropriate preprocessing (contour extraction for ruled line extraction, contour straight line approximation, grid extraction, etc.) is performed on pre-processing for teacher data composed of a request form and a pair of account information location or account information. Using the obtained teacher data, the learning target neural network is learned according to any appropriate learning algorithm (for example, Resnet, Xception, SENet, etc.).

  Furthermore, when extracting stamped seal imprint information, stamping is often performed in various directions, and it is desired to extract seal imprint information independent of the direction. For example, the learning device 60 performs appropriate preprocessing (for ruled line extraction) on teacher data composed of a pair of a filled-in type of bank transfer request form provided from the user terminal 50 and a stamp location. Contour extraction, contour straight line approximation, grid extraction, etc.), and using the pre-processed teacher data, according to any appropriate learning algorithm (eg, Resnet, Xception, SENet, SSD, etc.) Learn the network. When the stamping location is specified by the learned model, the form reading device 100 can extract an image at the specified stamping position, and the extracted image is extracted by the form reading device 100 or another collation device (not shown). It is possible to automatically collate the image thus obtained with the imprint information registered by the customer.

  The storage 70 stores a learned model used by the form reading device 100. For example, the learned model is learned by the learning device 60, and given information (for example, client code, location of account information, location of account information, location of imprint information, Machine learning model such as a neural network that extracts

  As described in detail below, the form reading device 100 responds to a start instruction of the form reading process from the user terminal 50, and the form reading device 100 reads the form read by a scanner (not shown) or the like. A form reading process is performed on the image information generated from the.

  The form reading device 100 and the learning device 60 are typically realized by a server, and may have a hardware configuration as shown in FIG. 2, for example. That is, the form reading device 100 and the learning device 60 include a drive device 101, an auxiliary storage device 102, a memory device 103, a CPU (Central Processing Unit) 104, an interface device 105, and a communication device 106, which are interconnected via the bus B. .

  Various computer programs including programs for realizing various functions and processes described below in the form reading device 100 and the learning device 60 may be provided by a recording medium 107 such as a CD-ROM (Compact Disk-Read Only Memory). When the recording medium 107 storing the program is set in the drive device 101, the program is installed from the recording medium 107 to the auxiliary storage device 102 via the drive device 101. However, the program need not always be installed on the recording medium 107, and may be downloaded from any external device via a network or the like. The auxiliary storage device 102 stores the installed program and also stores necessary files and data. The memory device 103 reads out and stores a program or data from the auxiliary storage device 102 when a program start instruction is issued. The CPU 104 that functions as a processor performs various functions and processes of the form reading device 100 and the learning device 60, which will be described later, according to various data such as programs stored in the memory device 103 and parameters necessary for executing the programs. Execute. The interface device 105 is used as a communication interface for connecting to a network or an external device. The communication device 106 executes various communication processes for communicating with an external device. However, the form reading device 100 and the learning device 60 are not limited to the hardware configuration described above, and may be realized by any other appropriate hardware configuration.

  Next, a form reader according to an embodiment of the present invention will be described with reference to FIG. FIG. 3 is a block diagram showing a functional configuration of the form reading apparatus according to one embodiment of the present invention.

  As shown in FIG. 3, the form reading device 100 includes an interface unit 110 and an extraction unit 120.

  When acquiring the filled-out form to be extracted, the interface unit 110 sends the form to the extracting unit 120 and acquires predetermined information from the extracting unit 120. More specifically, the interface unit 110 acquires image information of a plurality of types of forms generated by scanning the form with a scanner or the like, and passes the acquired image information to the extraction unit 120. After the extraction unit 120 performs the extraction process on the image information of the form using the learned model, the interface unit 110 acquires the information acquired from the extraction unit 120 and transmits the information to the user terminal 50, for example. Upon receiving the information, the user terminal 50 can store and display the predetermined information extracted from the form.

  The extracting unit 120 extracts predetermined information from the filled-out form using the learned model. Specifically, the extraction unit 120 inputs image information of a plurality of types of forms provided from the interface unit 110 to a machine learning model such as a neural network trained by the learning device 60, and is output from the trained model. And obtains predetermined information on the form. For example, when the form is a fund transfer request form, the extracting unit 120 inputs the image information of the fund transfer request form provided from the interface unit 110 to the trained model, and sets the entrustor code, account, Predetermined information such as information location, account information (account number, etc.), imprint location, imprint information, etc. is obtained as an output. The extracting unit 120 transmits the information thus obtained to the user terminal 50.

  In one embodiment, the learned model is an image feature extraction model that extracts an image feature amount from an image of a form, a ruled line feature extraction model that extracts a ruled line feature amount from a ruled line previously described in the form, and an image feature amount. And a target information extraction model for extracting predetermined information from the ruled line feature amount. Specifically, as shown in FIG. 4, the trained model has a multimodal neural network structure for processing two types of input, a form image and a ruled line printed on the form, respectively. And a ruled line feature amount extraction model for extracting a ruled line feature amount from a ruled line image. The learning device 60 generates a form image and a ruled line image from a plurality of types of filled out forms as preprocessing, inputs the form image to an image feature extraction model, acquires an image feature amount, and extracts the ruled line image from the ruled line feature extraction. Input to the model to obtain ruled line feature values. Note that the ruled line image does not necessarily need to be generated from a completed form, but may be generated from a corresponding blank form. The learning device 60 inputs the acquired image feature amount and ruled line feature amount to the target information extraction model, and performs predetermined information (for example, entrustor code, account information location, account information, imprint location, imprint information, and the like). To get output information. The learning device 60 learns these models by comparing the acquired output information with the teacher data and updating the parameters of the image feature extraction model, ruled line feature extraction model, and target information extraction model according to the error. Further, the learned model is not limited to the extraction of the ruled line portion, and may include an area extraction model for extracting a predetermined graphic area such as a rectangular area described in a form in advance.

  Further, in one embodiment, the learned model may include a form type determination model for determining a form type from a form image, and a target information extraction model for each form type. More specifically, the learning device 60 learns a form type determination model based on teacher data composed of pairs of completed forms and a type of the form, and further generates a target information extraction model for each form type. Each target information extraction model is learned using teacher data prepared and prepared as a pair of a filled-out form of a specific type and predetermined information. When the extracting unit 120 obtains the filled-out forms, first, the form is input to the form type determination model to specify the form type in order to determine the form type. After that, the extraction unit 120 inputs the completed form into the target information extraction model corresponding to the specified form type, and extracts predetermined information (account information location, account information, imprint location, imprint information, etc.). I do. According to the present embodiment, it is necessary to prepare a target information extraction model for each form type. For example, when there is a change in the form or layout of the form, the form type determination model and the target of the changed form Only the information extraction model needs to be re-learned, which is excellent in maintainability, and is a target information extraction model specialized in a specific format and layout. Therefore, improvement in extraction accuracy is expected.

  Next, a model learning process according to an embodiment of the present invention will be described with reference to FIG. FIG. 5 is a flowchart illustrating a model learning process according to an embodiment of the present invention. The model learning process is executed by the learning device 60, more specifically, the processor of the learning device 60.

  As shown in FIG. 5, in step S101, the learning device 60 sets the entered plural forms and information to be extracted (for example, entrustor code, account information location, account information, imprint location, imprint information). ) Is acquired from the user terminal 50 or the like. For example, the learning device 60 learns a model for each information type to be extracted.

  In step S102, the learning device 60 inputs the completed form for training to the learning target model. The learning device 60 may perform appropriate pre-processing on the completed form for training, and input the pre-processed form to the learning target model.

  In step S103, the learning device 60 acquires output data from the learning target model.

  In step S104, the learning device 60 compares the information of the extraction target for training with the output data, and updates the parameters of the learning target model according to back propagation or the like based on the error between them.

  The learning device 60 executes the processing of steps S101 to S104 on each teacher data until the termination condition is satisfied, and stores the finally acquired learning target model in the storage 70 as a learned model. The termination condition may be that a predetermined number of teacher data has been processed, that the error has become equal to or less than a predetermined threshold, that the error has converged, or the like.

  Next, a form reading process according to an embodiment of the present invention will be described with reference to FIG. FIG. 6 is a flowchart showing a form reading process using a learned model according to an embodiment of the present invention. The form reading process is executed by the form reading device 100, more specifically, by the processor of the form reading device 100.

  As shown in FIG. 6, in step S201, the form reading apparatus 100 inputs a filled-out form to be extracted to a learned model.

  In step S202, the form reading device 100 acquires predetermined information from the learned model. The information to be acquired may be all or a part of information entered in the form according to the learned model. The form reading device 100 transmits the extracted information to the user terminal 50 or the like.

  As described above, the embodiments of the present invention have been described in detail. However, the present invention is not limited to the specific embodiments described above, and various modifications may be made within the scope of the present invention described in the appended claims.・ Change is possible.

10 form reading system 50 user terminal 60 learning device 70 storage 100 form reading device

Claims (5)

  A form reader having an extraction unit for extracting predetermined information from a filled-out form using a learned model. The form is a fund transfer request form,
The form reading device according to claim 1, wherein the predetermined information is information necessary for executing an account transfer.   The form reading apparatus according to claim 1, wherein the learned model is learned for each predetermined information to be extracted. The trained model is
A first sub-model for extracting an image feature amount from the image of the form,
A second sub-model for extracting a ruled line feature from a ruled line previously described in the form;
A third sub-model for extracting the predetermined information from the image feature amount and the ruled line feature amount;
The form reading apparatus according to claim 1, further comprising:   The form reading device according to any one of claims 1 to 4, further comprising an interface unit that, when acquiring a filled-out form to be extracted, sends the form to the extracting unit and acquires the predetermined information from the extracting unit. apparatus.

Images