JP6590355B1 - Learning model generation device, character recognition device, learning model generation method, character recognition method, and program - Google Patents

Abstract

An object of the present invention is to improve the accuracy of recognition processing of a handwritten character string entered in a form. A learning model generation device includes a character string based on a corpus in which one or more words that can be entered in a handwritten character area of a form are registered, and a data set DS1 of a handwritten character image for each character. A learning data generation unit 12 that generates learning data including an image and a correct answer label, and a learning model generation unit 13 that generates a learning model by first learning using the learning data. [Selection] Figure 3

Description

  The present invention relates to a learning model generation device, a character recognition device, a learning model generation method, a character recognition method, and a program.

  Conventionally, digital data converted into a predetermined character code is generated by performing optical character recognition processing, that is, ОCR (Optical Character Recognition) processing, on image data obtained by reading a form in which handwritten characters are written by an image scanner or the like. Techniques are known.

  For example, Patent Document 1 discloses a system and related services for recognizing characters in documents such as handwritten characters and printed characters, and images such as videos and photographs. More specifically, Patent Document 1 discloses a character image input receiving unit that receives an input of a sample character image, a character component extraction that extracts a character component based on the sample character image, and a pseudo character model based on the character component. There is described a character identification system including a pseudo character model generation unit that generates a character identification pattern and an identification dictionary generation that generates a character identification pattern based on the pseudo character model and generates an identification dictionary.

Japanese Patent Application Laid-Open No. 2015-069256

  In the conventional system described in Patent Document 1, it is described that a character registered as an external character or a new character image is learned based on a small number of sample images and the character is recognized with higher accuracy. . However, the character recognition technique described in Patent Document 1 is for individually recognizing each character, and is not for reading a character string composed of a plurality of characters with high accuracy.

  Accordingly, some aspects of the present invention have been made in view of such circumstances, and a learning model generation device, a character recognition device, and a learning model generation method that improve the accuracy of recognition processing of a handwritten character string entered in a form. An object of the present invention is to provide a character recognition method and a program.

  A learning model generation apparatus according to an aspect of the present invention is based on a database in which one or more words that can be entered in a handwritten character area of a form are registered, and a data set of handwritten character images in units of characters. A learning data generation unit that generates learning data including a sequence image and a correct label, and a learning model generation unit that generates a learning model by first learning using the learning data.

  A character recognition device according to an aspect of the present invention is a character recognition device that recognizes handwritten characters entered in a form, based on an image data acquisition unit that acquires image data of the form, and the acquired image data, Handwritten characters using a learning model having a network structure in which one or more handwritten character regions including a character string written in handwritten characters are specified, and a first neural network and a second neural network are combined. And a character recognition unit for recognizing the contents of the character string entered in the area.

  A learning model generation method according to an aspect of the present invention is a learning model generation method executed by a computer that generates a learning model, and includes a database in which one or a plurality of words that can be written in a handwritten character area of a form are registered A step of generating learning data including a character string image and a correct answer label based on a data set of handwritten character images in units of one character, and a step of generating a learning model by first learning using the learning data And including.

  A character recognition method according to an aspect of the present invention is a character recognition method executed by a computer for recognizing handwritten characters entered in a form, based on the step of obtaining image data of the form, and based on the obtained image data Using a learned model having a network structure in which one or a plurality of handwritten character regions including a character string written with handwritten characters is identified and the first neural network and the second neural network are combined. Recognizing the contents of the character string entered in the area.

  A program according to an aspect of the present invention is a program that uses a computer based on a database in which one or more words that can be entered in a handwritten character area of a form are registered, and a data set of handwritten character images for each character. It functions as a learning data generation unit that generates learning data including a sequence image and a correct answer label, and a learning model generation unit that generates a learning model by first learning using the learning data.

  A program according to an aspect of the present invention is a computer that recognizes handwritten characters entered in a form, an image data obtaining unit that obtains image data of the form, and handwritten characters based on the obtained image data. Using a learned model having a network structure in which one or a plurality of handwritten character regions including a character string are specified and a first neural network and a second neural network are combined, the handwritten character region is filled It functions as a character recognition unit that recognizes the contents of a character string.

  In the present invention, the “part” does not simply mean a physical means, but includes a case where the function of the “part” is realized by software. Also, even if the functions of one “unit” or device are realized by two or more physical means or devices, the functions of two or more “units” or devices are realized by one physical means or device. May be.

  ADVANTAGE OF THE INVENTION According to this invention, the precision of the recognition process of the handwritten character string entered in the form can be improved.

It is a schematic block diagram (system block diagram) of the character recognition apparatus which concerns on 1st Embodiment. It is a figure showing an example of securities concerning a 1st embodiment. It is a conceptual diagram which shows an example of the data generation process for learning which concerns on 1st Embodiment, and a learning model generation process. It is a figure which shows an example of the address corpus concerning 1st Embodiment. It is a conceptual diagram which shows an example of the learning model reinforcement | strengthening (update) process which concerns on 1st Embodiment. It is a conceptual diagram which shows an example of the layout information which concerns on 1st Embodiment. It is a flowchart which shows an example of the character recognition process which concerns on 1st Embodiment. It is a flowchart which shows an example of the learning model production | generation process which concerns on 1st Embodiment. It is a figure which shows an example of the form on which the watermark based on 2nd Embodiment was printed. It is a figure which shows an example of the data for learning which superimposes at least one part of the watermark printed on a form on the character string image based on 2nd Embodiment. It is a schematic block diagram (system block diagram) of the character recognition apparatus which concerns on 3rd Embodiment. It is a schematic block diagram (system block diagram) of the character recognition apparatus and learning model production | generation apparatus which concern on 4th Embodiment. It is a schematic block diagram (system block diagram) of the character recognition apparatus which concerns on 5th Embodiment, and an external device. It is a figure which shows an example of the hardware constitutions of the computer which concerns on embodiment of this invention. It is a schematic block diagram (system block diagram) which shows the modification of the character recognition apparatus which concerns on 1st Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The following embodiments are exemplifications for explaining the present invention, and are not intended to limit the present invention only to the embodiments. The present invention can be variously modified without departing from the gist thereof. Furthermore, in each drawing, the same components are denoted by the same reference numerals as much as possible, and redundant description is omitted.

<First Embodiment>
FIG. 1 is a schematic configuration diagram (system configuration diagram) of a character recognition device according to a first embodiment of the present invention. As shown in FIG. 1, the character recognition device 100A is a device that recognizes handwritten characters entered in a form, and is an information processing device such as a server. The character recognition device 100A may be another information processing device such as a laptop or a notebook computer. The character recognition device 100A exemplarily includes an information processing unit 1 that performs information processing for recognizing handwritten characters entered in a form, a corpus 3 as a dictionary database (DB), and a handwritten character data set for each character. It comprises DB5, handwritten character data set DB7 for each character string image, and layout information DB9. Note that the character recognition device 100A may recognize characters other than the handwritten characters entered in the form. Further, at least one of the corpus 3, the handwritten character data set DB5, the handwritten character data set DB7, or the layout information DB9 may be configured as a device separate from the character recognition device 100A or as a database.

  “Form” means a document in which a character string is written, and is a general term for books and slips. The form includes documents such as securities handled by securities companies, application forms, or contracts, for example.

  FIG. 2 is a diagram illustrating an example of a security (form) according to the first embodiment. As shown in FIG. 2, the security C1 is a car insurance policy of a specific insurance company “XX Insurance Co., Ltd.”.

  The securities C1 includes, for example, an address field 20 (handwritten character region relating to an address) and a name field 22 as fields (handwritten character regions) in which handwritten characters are entered. These fields are examples of fields for recognizing characters by the character recognition device 100A, and the securities C1 may have other fields in which characters are described. For example, the securities C1 may have an insurance period field (not shown) for hand-filling the insurance start date and the insurance maturity date included in the “contract details”. Further, the securities C1 includes a chassis number field (not shown) for handwritten entry of the chassis number and a registration number field (not shown) for entering the registration number, which are included in the "contracted car". And the like. Further, the character recognition device 100A may recognize handwritten characters in other fields. It should be noted that handwritten characters are not necessarily recognized for all the above-described fields.

  Returning to FIG. 1, for example, the information processing unit 1 functionally includes an image data acquisition unit 11, a learning data generation unit 12, a learning model generation unit 13, a learning model update unit 14, a region specification unit 15, and characters. The recognition unit 16 is included.

  The above-described units of the information processing unit 1 can be realized by using a storage area such as a memory or a hard disk, or by executing a program stored in the storage area by a processor. Further, the corpus 3 and the DBs 5, 7, and 9 of the character recognition device 100A can be realized by being executed by a processor.

  The image data acquisition unit 11 acquires image data of the securities C1. Further, the image data acquisition unit 11, for example, the image data acquisition unit 11 captures image data generated by imaging with an imaging device such as a camera which is an example of the input / output interface 44 described later with reference to FIG. 14. You may get it.

  As shown in FIG. 15, the image data acquisition unit 11 acquires image data generated by imaging the securities C <b> 1 with an external device 50 including an imaging device such as a camera via a predetermined communication network N. Also good. The communication network N is a communication line or a communication network related to information processing including the Internet, for example, and its specific configuration is configured so that data can be transmitted and received between the character recognition device 100A and the external device 50. If it is done, it is not particularly limited.

  FIG. 3 is a conceptual diagram illustrating an example of learning data generation processing and learning model generation processing according to the first embodiment. As shown in FIGS. 1 and 3, the learning data generation unit 12 includes a corpus 3 in which one or more words that can be entered in one or more handwritten character areas in the securities C1 are registered, and a handwritten character data set DB5. The learning data including the character string image and the correct answer label is generated based on the data set DS1 of the handwritten character image in units of one character recorded in (1).

  FIG. 4 is a diagram illustrating an example of an address corpus according to the first embodiment. As shown in FIG. 4, in the corpus 3 shown in FIGS. 1 and 3, for example, addresses in which possible combinations of prefecture names, city names, area names, and building names are layered and registered. Corpus is included. In other words, the name of a city belonging to the prefecture is included in the lower hierarchy of each prefecture name. Similarly, the lower level of each city name includes the name of the area belonging to the city. Further, the old name and the current name may be associated with each other and registered in the address corpus with respect to the name of the prefecture, city, town and village where the administrative change has occurred.

  The corpus 3 shown in FIGS. 1 and 3 may include various corpora specialized in name, car name, occupation, insurance period, chassis number, registration number, etc. in addition to the address corpus specialized in address. Good.

As shown in FIGS. 3 and 4, the learning data generation unit 12 extracts text information CL1 of one or more words that can be entered in one or more fields in the securities C1 from the corpus 3. In the example shown in FIG. 3, for example, address text information “Tokyo” is extracted from the address corpus. Next, the handwritten character image of each character included in the text information CL1 extracted from the corpus 3 is read from the handwritten character data set DB 5 to generate a handwritten character string image CSI1 “Tokyo”. Here, as shown in FIG. 3, the handwritten character data set DB 5 stores a plurality of corresponding handwritten character images in association with each character (one character) such as hiragana, katakana, and kanji. Specifically, a plurality of handwritten character images “A” (that is, images handwritten as “A”) are stored for the text information “A”. Similarly, for each character other than the text information “A”, the handwritten character data set DB5 stores a data set DS1 in which a plurality of handwritten character images (one character-by-character handwritten character image) are associated.
That is, for each character included in the text information CL1 extracted from the corpus 3, the learning data generation unit 12 reads a corresponding handwritten character image from the handwritten character data set DB5 and generates a handwritten character string image CSI1. To do. For example, when the text information CL1 is “Tokyo”, one handwritten character image “G” corresponding to the text information “G” is arbitrarily extracted from the handwritten character data set DB5. Subsequently, one handwritten character image “U” corresponding to the text information “U” is arbitrarily extracted from the handwritten character data set DB 5. Similarly for the remaining text information “ki”, “yo”, “u” and “to”, the corresponding handwritten character images “ki”, “yo”, “u” and “to” are respectively arbitrarily selected from the handwritten character data set DB 5. Extract one. Then, the extracted handwritten character images “T”, “U”, “K”, “K”, “U”, and “G” are combined into one to generate a character string image CSI1 “Tokyo”. For the handwritten character string image CSI1 “Tokyo” generated in this way, learning data is generated with the text information CL1 “Tokyo” extracted from the corpus 3 as the correct answer label. When the handwritten character string image CSI1 is generated, each of the arbitrarily extracted handwritten character images “t”, “c”, “ki”, “c”, “c”, “g” is rotated, enlarged, reduced, moved. Alternatively, distortion may be added.

  In the handwritten character data set DB 5, as a data set of handwritten character images for each character, a set of hiragana or kanji text information and a plurality of handwritten character images corresponding to each of the hiragana or kanji text information. May be included. The handwritten character data set DB5 may include a data set of handwritten character images in units of foreign characters. For example, as a data set of handwritten character images in character units, alphabetic text information and a plurality of handwritten character images corresponding to the alphabetic text information may be included in a set.

  The learning model generation unit 13 generates a learning model by first learning using the learning data generated by the learning data generation unit 12. As illustrated in FIG. 3, the learning model generation unit 13 generates a learning model LM1 from a network structure including, for example, CRNN (Convolutional Recurrent Neural Network). The CRNN is, for example, a network structure in which a convolutional neural network (first neural network), that is, a CNN (Convolutional Neural Network) and a recurrent neural network (second neural network), that is, an RNN (Recurrent Neural Network) are combined. . In CNN, the feature-value map regarding the handwritten character string contained in the 1 or several field in the securities | curds C1 is calculated. The RNN is a neural network that can handle time-series data such as moving images and sounds, and has a recursive structure, so that prediction including past information is possible. Using RNN, a character string index is calculated based on the context of a plurality of continuous feature data obtained from the feature map. The network structure may adopt a configuration other than the above. As the neural network, neural networks other than CNN and RNN may be adopted.

  According to this configuration, the learning model generation unit 13 generates a learning model from a network structure including CRNN, and thus can recognize a handwritten character string with high accuracy.

  As described above, the learning model generation unit 13 generates an address CRNN learning model. The learning model generation unit 13 may generate another type of CRNN learning model. For example, when the learning data generation unit 12 generates learning data related to name, insurance period, chassis number, registration number, etc., the learning model generation unit 13 includes name, vehicle name, occupation, insurance period, chassis number. And a first learning using each of the learning data relating to the registration number, etc., a CRNN learning model relating to the name, the insurance period, the chassis number, the registration number, etc. may be generated.

  After the first learning shown in FIG. 3, the learning model update unit 14 reinforces (updates) the generated learning model by second learning using a character string image cut out from the image data of the securities C1 as learning data. )

  FIG. 5 is a conceptual diagram illustrating an example of learning model reinforcement (update) processing according to the first embodiment. As shown in FIG. 5, the learning model update unit 14 shown in FIG. 1 uses a plurality of character string images such as addresses and names extracted from the image data of a plurality of securities C1 as learning data, and sets a handwritten character data set. Store in DB7. The handwritten character data set DB 7 stores, for example, a data set DS3 of handwritten character images in units of character string images including a plurality of handwritten character string images “Tokyo”. The learning model update unit 14 uses, for example, the existing learning model generated in the first learning shown in FIG. 3, for example, the handwritten character string image “Tokyo” (“U” is unclear) included in the handwritten character data set DB 7. Based on the above, the text information “Tokyo” is generated (inferred). This is because the image “U” is misrecognized as “K” because “U” of the handwritten character string image “Tokyo” is unclear. In this case, for example, the user who operates the character recognition device 100A shown in FIG. 1 may execute manual correction for correcting the misrecognized text information “Tokyo” to the text information “Tokyo”.

  As described above, the learning model update unit 14 generates (infers) text information based on the handwritten character string image using the existing learning model generated in the first learning, and when it is erroneously recognized, The text information manually corrected by is given as a correct label of the handwritten character string image. On the other hand, if no erroneous recognition is made, manual correction is not executed, and the generated (inferred) text information is assigned as a correct label of the handwritten character string image. Thus, in the second learning, an annotation is generated semi-automatically. That is, learning data including a handwritten character string image and a correct answer label corresponding thereto, that is, in this example, “Tokyo” of text information corresponding to the handwritten character string image “Tokyo” is generated. The learning model update unit 14 can reinforce the learning model by adding newly generated learning data to the existing learning model.

  According to this structure, the learning model update part 14 extracts the handwritten character string of each item (name, address, name, etc.) from the image data of the securities C1, and gives a correct answer label to them. Therefore, the learning model generated in the first learning can be strengthened by performing the second learning on the handwritten character string to which these correct labels are assigned.

  Returning to FIG. 1, the region specifying unit 15 specifies one or a plurality of fields including a character string written in handwritten characters on the certificate C <b> 1 based on the image data of the certificate C <b> 1 acquired by the image data acquiring unit 11. Various methods can be used for specifying the field in the security C1, but as an example, a method using layout information for specifying the field in the security C1 will be described below.

  As shown in FIG. 1, the character recognition device 100A further includes a layout information DB 9 (recording unit) that records layout information for specifying a field in association with a predetermined position in the securities C1.

  FIG. 6 is a conceptual diagram showing an example of layout information according to the first embodiment. As shown in FIG. 6, the layout information stores, for each securities template ID, the field names of a plurality of fields and the start position and end position of the fields in association with each other. Here, the field name is an example of information specifying which of a plurality of fields.

  In the example of FIG. 6, for the securities template ID “001”, the position of the field name “address” is represented by the coordinates of the start point (X21, Y21) and the coordinates of the end point (X22, Y22). Thereby, the field of the field name “address” is a rectangular area designated by the start point and the end point. These coordinates are preferably positions when the entire security C1 is normalized to a predetermined size. However, the field position designation method is not limited to the example shown in FIG. 6, and other methods may be used.

  The layout information is stored in the layout information DB 9 prior to the character recognition process of the character recognition device 100A. When a securities in a new format is issued, it is preferable that layout information about the securities is added to the layout information DB 9 by the user of the character recognition device 100A or the like.

As described above, the layout information stores information on positions for specifying each of a plurality of fields for each securities template. The area specifying unit 15 specifies a field based on the layout information. According to this configuration, for example, even in a plurality of securities having different layouts, the position of each field such as the address field 20 and the name field 22 can be specified.

  The method for specifying the field in the securities C1 is not limited to the above. For example, an example of a specific method that does not use the above-described securities template will be described as a third embodiment.

  The character recognition unit 16 recognizes the contents of the character string entered in the handwritten character area using the generated learning model or the enhanced (updated) learning model. The character recognizing unit 16 recognizes the content of the character string entered in the field of the security C1 using, for example, a learning model having a CRNN in which CNN and RNN are combined. According to this configuration, since the character recognition unit 16 recognizes a handwritten character string using a network structure in which CNN and RNN are combined, it can recognize the handwritten character string with high accuracy.

(Character recognition processing)
An example of character recognition processing according to the first embodiment of the present invention will be described with reference to FIGS. FIG. 7 is a flowchart illustrating an example of character recognition processing according to the first embodiment.

  As shown in FIG. 7, the image data acquisition unit 11 shown in FIG. 1 acquires the image data of the securities C1 shown in FIG. 2 (step S1). The learning data generation unit 12 is a corpus 3 in which one or a plurality of words that can be entered in the handwritten character area of the certificate C1 is registered, and data of handwritten character images in character units recorded in the handwritten character data set DB5. Based on the set DS1, learning data including a character string image and a correct label is generated (step S3). The learning model generation unit 13 generates a learning model (step S5). The learning model generation process and update process will be described later with reference to FIG. Based on the image data acquired by the image data acquisition unit 11, the region specifying unit 15 specifies one or a plurality of fields including a character string written with handwritten characters on the securities C1 (step S7). The character recognition unit 16 uses the generated learning model or the strengthened (updated) learning model, and the character string written in one or more fields including the character string written in handwritten characters on the securities C1. Is recognized (step S9).

  FIG. 8 is a flowchart showing an example of the learning model generation process (step S5 in FIG. 7) according to the first embodiment. As illustrated in FIG. 8, the learning model generation unit 13 generates a learning model through first learning using the learning data generated by the learning data generation unit 12 (step S51). Next, after the first learning, the learning model update unit 14 updates the learning model by the second learning using the character string image cut out from the image data of the securities C1 as learning data (step S53).

  As mentioned above, according to 1st Embodiment of this invention, a character string image and a correct answer label are based on corpus 3 and data set DS1 of the handwritten character image of 1 character unit currently recorded on handwritten character data set DB5. Generate learning data including. A learning model is generated by first learning using the generated learning data. Therefore, it is possible to recognize the contents of the character strings entered in one or more fields in the securities C1 using the learning model generated by the first learning. Therefore, it is possible to improve the accuracy of the recognition process of the handwritten character string entered in the securities C1.

Second Embodiment
A learning model generation process and a learning model update process of the second embodiment will be described with reference to FIGS. In the second embodiment, learning data is obtained by superimposing at least a part of a watermark printed on a form on a character string image included in the handwritten character data sets DB5 and DB7 shown in FIGS. In the point which produces | generates, it differs from 1st Embodiment in which the watermark is not superimposed on the character string image contained in handwritten character data set DB5, DB7 shown in FIG.1, FIG3 and FIG.5. In the second embodiment, the character string images included in the handwritten character data sets DB5 and DB7 shown in FIG. 1, FIG. 3 and FIG. This is different from the first embodiment in which noise is not superimposed on the character string images included in the handwritten character data sets DB5 and DB7 shown in FIGS. Below, a different point from 1st Embodiment is demonstrated especially.

FIG. 9 is a diagram illustrating an example of a security printed with a watermark according to the second embodiment. As shown in FIG. 9, the security C3 is, for example, an automobile insurance policy, and a watermark W “copy” is printed on the security C3. FIG. 10 is a diagram illustrating an example of learning data in which at least a part of the watermark printed on the securities C3 is superimposed on the character string image according to the second embodiment.
As shown in FIG. 10A, in the one-character handwritten character data set DB5 shown in FIGS. 1 and 3, for example, a plurality of one-character handwritten character images “ki” including at least a part of a watermark are included. The pattern is stored. The handwritten character data set DB5 is not limited to this, and a plurality of patterns may be stored for each of the handwritten character images “a”. The learning data generation unit 12 shown in FIG. 1 reads out a plurality of patterns of handwritten character images “a”... “N” including at least a part of the watermark from the handwritten character data set DB 5 at random. Is generated.

  The handwritten character data set DB 5 includes a plurality of handwritten characters corresponding to each of hiragana or kanji text information and hiragana or kanji text information as a data set of handwritten character images including at least a part of the watermark. A character image may be included as a set. In addition, the handwritten character data set DB5 may include a data set of handwritten character images for each character in a foreign language including at least part of the watermark. For example, as a data set of handwritten character images in character units, alphabetic text information and a plurality of handwritten character images corresponding to the alphabetic text information may be included in a set.

As shown in FIG. 10B, in the handwritten character data set DB 7 for each character string image shown in FIGS. 1 and 5, for example, a handwritten character string image “Tokyo *” including at least a part of the watermark is included. ** ... "is stored. Further, as shown in FIG. 10C, the handwritten character data set DB7 in character string image units shown in FIGS. 1 and 5 includes, for example, a handwritten character string image “in character string units including at least a part of a watermark”. “Tokyo” (part of the handwritten character string image shown in FIG. 10B) may be stored.
The learning model updating unit 14 according to the present embodiment is a character string image cut out from the image data of the security C3 after the first learning shown in FIG. 3, and is a character string on which at least a part of the watermark is superimposed. The image is read from the handwritten character data set DB7. The learning model update unit 14 reinforces (updates) the generated learning model by second learning using, for example, the read character string image on which at least a part of the watermark is superimposed as learning data. .

  Note that the learning data used in at least one of the learning model generation process and the learning model reinforcement process is a character string image in which at least a part of the watermark printed on the securities is superimposed on the character string image. In addition, it may include noise superimposed on securities.

  As described above, according to the second embodiment of the present invention, at least one part of the watermark printed on the form is superimposed on the character string image in at least one of the learning model generation process and the learning model reinforcement process. Is generated as learning data. Therefore, robustness is improved in the character string image recognition process in the form on which the watermark is printed.

  Further, in at least one of the learning model generation process and the learning model reinforcement process, a character string image in which noise in the form is superimposed is generated as learning data. Therefore, the robustness is improved in the character string image recognition process in the form on which noise is superimposed.

<Third Embodiment>
A character recognition device according to the third embodiment will be described with reference to FIG. The character recognition device 100B according to the third embodiment performs processing such as extracting an item area including an item name described in a form and assigning an attribute to the item area in the process of specifying a form field. The first embodiment is different from the third embodiment in that these processes of the third embodiment are not performed and the form fields are specified with reference to the layout information shown in FIG. Below, a different point from 1st Embodiment is demonstrated especially.

  FIG. 11 is a schematic configuration diagram (system configuration diagram) of the character recognition device according to the third embodiment. As shown in FIG. 11, the character recognition device 100B does not include the layout information DB 9 shown in FIG. 1 as compared with the character recognition device 100A according to the first embodiment shown in FIG. For example, an item extracting unit 151 and an attribute assigning unit 152 are further provided.

  The area specifying unit 15 uses, for example, an item area including an item name such as “name” and “address” described in the securities as an extraction target and assigns an attribute to the item area using a predetermined neural network. An item extraction unit 151 for extraction is provided. The item extraction unit 151 extracts, for example, item areas included in the image data of the securities with attribute classifications, by using, as extraction targets, item areas including the item names printed in print on the securities. For example, when there are image areas such as “name” and “address” in the image data of securities, each image area is extracted as an item area, and “name” or “address” is assigned to each item area. Such attributes are added. The item area is extracted using an object detection algorithm based on deep learning. Further, the attribute of the extracted item region is classified with reference to a predetermined learning model constructed based on this algorithm. In addition, for the classified attributes, the classification accuracy may be calculated and output.

  The area specifying unit 15 associates an item area with a field (handwritten character area) located near the item area based on the position and attribute of the item area in the image data of the security, and An attribute assigning unit 152 that assigns the attribute of the item area is provided. The attribute assigning unit 152 analyzes the layout of the securities image based on the position of the item area in the image data of the securities and the attributes thereof, and identifies where the information regarding which attribute is entered. Specifically, one of the attributes classified by the item extraction unit 151 is assigned to each field in the securities. Basically, in a security image data, when a certain item area and a certain handwritten character area are close to each other, that is, when the distance between them is equal to or smaller than a predetermined threshold value, they are associated with each other. The attribute of the item area is assigned to this field (handwritten character area). For example, if a field exists in the vicinity of an item area having the attribute “name”, the attribute “name” is assigned to this field. Further, specific correspondence rules between item areas and fields are set and defined in advance in a predetermined correspondence rule table or the like.

  As described above, according to the third embodiment, the item areas included in the image data of securities and their attributes are acquired. From these pieces of information, it is possible to specify what information is written at which position in the form image. This makes it possible to perform layout analysis even for unknown securities that are not registered in advance in the character recognition device.

<Fourth embodiment>
A character recognition device and a learning model generation device according to the fourth embodiment will be described with reference to FIG. The character recognition device 100C and the learning model generation device 200 according to the fourth embodiment shown in FIG. 12 are configured by separating the components included in the character recognition device 100A according to the first embodiment shown in FIG. . The character recognition device 100C uses the learning model generated by the learning model generation device 200, for example, a learning model having a network structure in which the first neural network and the second neural network are combined, and the contents of the character string entered in the field. Recognize Further, the character recognition device 100C and the learning model generation device 200 may be configured to be able to transmit and receive data via the communication network N. However, the present invention is not limited to this, and the learning model generated by the learning model generation device 200 is stored in the main recording device of the character recognition device 100C by any means. Similarly, for the character recognition device 100B according to the third embodiment shown in FIG. 11, the character recognition device and the learning model generation device may be configured by separating the components included in the character recognition device 100B. In addition, the character recognition device and the learning model generation device may be configured to be able to transmit and receive data via the communication network N, in which the components included in the character recognition device 100B are separated.

  As described above, according to the fourth embodiment, the character recognition device 100A according to the first embodiment or the character recognition device 100C according to the third embodiment can configure a character recognition device and a learning model generation device which are separate devices. is there.

<Fifth Embodiment>
A character recognition device and an external device according to the fifth embodiment will be described with reference to FIG. FIG. 13 is a schematic configuration diagram (system configuration diagram) of a character recognition device and an external device according to the fifth embodiment. As shown in FIG. 13, the character recognition device 100 </ b> C according to the fifth embodiment may acquire image data generated by the external device 50 via a predetermined communication network N.

  As described above, according to the fifth embodiment, the character recognition device 100 </ b> C can acquire the image data generated by the external device 50 and execute the character recognition process based on the acquired image data.

  FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer according to the embodiment of the present invention. 14, character recognition device 100A shown in FIGS. 1 and 15, character recognition device 100B shown in FIG. 11, character recognition device 100C shown in FIGS. 12 and 13, learning model generation device 200 shown in FIG. 12, and An example of the hardware configuration of a computer that can be used to configure the external device shown in FIGS. 13 and 15 will be described.

  As shown in FIG. 14, the computer 40 mainly includes a processor 41, a main recording device 42, an auxiliary recording device 43, an input / output interface 44, and a communication interface 45 as hardware resources. These are connected to each other via a bus line 46 including an address bus, a data bus, a control bus, and the like. An interface circuit (not shown) may be interposed between the bus line 46 and each hardware resource as appropriate.

  The processor 41 controls the entire computer. The processor 41 corresponds to, for example, the information processing unit 1 illustrated in FIGS. 1 and 11. The main recording device 42 provides a work area to the processor 41 and is a volatile memory such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The auxiliary recording device 43 is a non-volatile memory such as an HDD, an SSD, or a flash memory that stores software programs and data. The program, data, and the like are loaded from the auxiliary recording device 43 to the main recording device 42 via the bus line 46 at an arbitrary time. The auxiliary recording device 43 corresponds to, for example, the corpus 3, the handwritten character data set DB5, the handwritten character data set DB7, and the layout information DB9 shown in FIG. The auxiliary recording device 43 corresponds to, for example, the corpus 3, the handwritten character data set DB5, and the handwritten character data set DB7 shown in FIG.

  The input / output interface 44 performs one or both of presenting information and receiving input of information, and includes a camera, a keyboard, a mouse, a display, a touch panel display, a microphone, a speaker, a temperature sensor, and the like. The communication interface 45 is connected to the communication network N shown in FIGS. 1, 11, and 12, and transmits and receives data via the communication network N. The communication interface 45 and the communication network N can be connected by wire or wireless. The communication interface 45 may also acquire information relating to the network, for example, information relating to Wi-Fi access points, information relating to communication carrier base stations, and the like.

  It will be apparent to those skilled in the art that the computer 40 can function as a desired means, execute a desired step, and realize a desired function by cooperating with the hardware resources and software exemplified above. .

  In addition, each said embodiment is for making an understanding of this invention easy, and does not limit this invention and interpret it. The present invention can be changed / improved without departing from the gist thereof, and the present invention includes equivalents thereof. Further, the present invention can form various disclosures by appropriately combining a plurality of constituent elements disclosed in the respective embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements may be appropriately combined in different embodiments.

DESCRIPTION OF SYMBOLS 1,1A, 1B ... Information processing part, 3 ... Corpus, 5, 7 ... Handwritten character data set, 9 ... Layout information DB, 11 ... Image data acquisition part, 12 ... Learning data generation part, 13 ... Learning model generation part , 14 ... Learning model update unit, 15 ... Area specifying unit, 16 ... Character recognition unit, 41 ... Processor, 42 ... Main recording device, 43 ... Auxiliary recording device, 44 ... Input / output interface, 45 ... Communication interface, 46 ... Bus 50 ... External device, 100A, 100B, 100C ... Character recognition device, 151 ... Item extraction unit, 152 ... Attribute assignment unit, 200 ... Learning model generation device

Claims (12)

A database in which one or more words that can be entered in the handwritten character area of the form are registered;
A learning data generation unit that generates learning data including a character string image and a correct answer label based on a data set of handwritten character images in units of one character;
A learning model generation unit that generates a learning model by first learning using the learning data;
Learning model generation device. A learning model updating unit that updates the learning model by second learning using a character string image cut out from the image data of the form as learning data after the first learning;
The learning model generation apparatus according to claim 1. The learning data generation unit generates learning data by superimposing at least a part of a watermark printed on the form on the character string image.
The learning model generation apparatus according to claim 1 or 2. The learning data generation unit generates, as learning data, a noise image superimposed on the character string image.
The learning model production | generation apparatus as described in any one of Claims 1-3. The learning model generation unit
A learning model is generated from a network structure in which a first neural network and a second neural network are combined;
The first neural network comprises a convolutional neural network;
Calculating a feature amount map related to the character string included in the handwritten character region;
The second neural network is constituted by a recurrent neural network, and calculates a character string index from the feature map.
The learning model production | generation apparatus as described in any one of Claims 1-4. At least one area of the handwritten character area is a handwritten character area related to an address;
Wherein the handwritten character area co associated with the Pasu relates Address, state, city name, area name, or a combination comprising at least one name of a building name is registered,
The learning model production | generation apparatus as described in any one of Claims 1-5. An image data acquisition unit that acquires image data generated by imaging the form with an imaging device via a communication network;
The learning model production | generation apparatus as described in any one of Claims 1-6. An area specifying unit for specifying one or a plurality of handwritten character areas including a character string written with handwritten characters based on the image data of the form;
Using a learning model generated by the learning model generating unit according to any one of claims 1 to 7, and a character recognizing unit for recognizing the content of a character string written in the handwritten character region,
Character recognition device. A recording unit for recording layout information for specifying the handwritten character region in association with a predetermined position in the form;
The region specifying unit specifies the handwritten character region based on the layout information.
The character recognition device according to claim 8 . The region specifying unit includes:
An item extraction unit that extracts an item area including an item name described in the form with an attribute;
An attribute for associating the item region with the handwritten character region located in the vicinity of the item region and assigning the attribute of the handwritten character region based on the position of the item region in the image data of the form and the attribute An allocation unit;
The character recognition device according to claim 8 or 9 . A learning model generation method executed by a computer that generates a learning model,
A database in which one or more words that can be entered in the handwritten character area of the form are registered;
Generating learning data including a character string image and a correct label based on a data set of handwritten character images in character units;
Generating a learning model by first learning using the learning data;
including,
Learning model generation method. Computer
A database in which one or more words that can be entered in the handwritten character area of the form are registered;
A learning data generation unit that generates learning data including a character string image and a correct answer label based on a data set of handwritten character images in units of one character;
A learning model generation unit that generates a learning model by first learning using the learning data;
Program to make it work.