JP2020027598A - Character recognition device, character recognition method, and character recognition program - Google Patents

Abstract

To provide a character recognition device, a character recognition method and a character recognition program that are capable of constructing an image recognition model even for a document including multiple items so as to properly extract regions of the multiple items.SOLUTION: A character recognition device 10 to recognize characters written in a document comprises: a first setting unit 11a to set multiple regions for an image in the document on the basis of input; a second setting unit 11b to set multiple sub-regions included in any of the multiple regions on the basis of the input; a third setting unit 11c to set region types for the multiple regions and the multiple sub-regions on the basis of the input; and a generation unit 12 to generate learning data used for supervised learning of one or more image recognition models, including positions of the multiple regions in the image, positions of the multiple sub-regions in the multiple regions, and the region types.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a character recognition device, a character recognition method, and a character recognition program.

  Conventionally, a technique called OCR (Optical Character Recognition) for recognizing characters included in an image has been used. When the description location and size of characters in an image are unified, characters can be recognized with high accuracy by OCR, and character information can be extracted from the image.

  For example, Patent Document 1 below describes an image processing program that acquires character string information and position information from image data of a receipt. The image processing program described in Patent Literature 1 displays a marker at a position corresponding to position information, and performs one of a first process of adding a plurality of values or one of a plurality of values according to the type of item information. Execute the second process to be selected.

JP-A-2006-126356

  When the description location and size of characters in an image are unified, character information can be extracted from the image with high accuracy by the conventional OCR technology. However, in the case of a document image including a plurality of items, it has been difficult for the conventional OCR technology to appropriately recognize the characters described in the plurality of items.

  Therefore, it has been studied to extract regions of a plurality of items included in a document image and recognize characters by applying the OCR technique to each of the regions of the plurality of items. Here, an image recognition model such as a neural network may be used to extract regions of a plurality of items from a document image.

  However, the number of items included in the document may be larger than the number of regions that can be distinguished by the image recognition model, and the region of a plurality of items included in the document image may not be appropriately extracted.

  Therefore, the present invention provides a character recognition device, a character recognition method, and a character recognition program capable of constructing an image recognition model so as to appropriately extract regions of a plurality of items even in a document including a plurality of items. I will provide a.

  A character recognition device according to one aspect of the present invention is a character recognition device for recognizing characters described in a document, comprising: a first setting unit configured to set a plurality of regions in a document image based on an input; A second setting unit that sets a plurality of sub-regions included in any of the plurality of regions based on the input; A setting unit and a generation unit that generates learning data used for supervised learning of one or a plurality of image recognition models, including positions of a plurality of regions in an image, positions of a plurality of sub-regions in the plurality of regions, and types of regions. And.

  According to this aspect, even when the number of items included in the document is larger than the number of areas that can be distinguished by the image recognition model, the number of the plurality of areas is reduced by the number of areas that can be distinguished by the image recognition model. It is possible to set a smaller number, set a plurality of sub-areas included in any of the plurality of areas, and set an area corresponding to a plurality of items. This makes it possible to identify a plurality of areas and a plurality of sub-areas in a stepwise manner and to construct an image recognition model for identifying the type of those areas. Even if the document includes a plurality of items, a plurality of areas can be identified. An image recognition model can be constructed so as to appropriately extract a region of an item.

  In the above aspect, the generation unit may generate a plurality of types of learning data used for supervised learning of one or a plurality of image recognition models.

  According to this aspect, by setting a plurality of regions, a plurality of sub-regions, and a type of the region once, a plurality of types of learning data used for supervised learning of one or a plurality of image recognition models are collectively generated. Thus, a plurality of types of learning data can be efficiently generated.

  In the above aspect, the generation unit may include a first type of learning data in which the outlines of the plurality of regions and the plurality of sub-regions are expressed in a different manner for each type of region, and the plurality of regions and the plurality of sub-regions. And the second type of learning data painted in a different manner for each type.

  According to this aspect, it is possible to generate a first image recognition model that overwrites an image with a bounding box that surrounds a plurality of regions and a plurality of sub-regions in a distinguishable manner, using the first type of learning data. In addition, a second image recognition model that outputs an image in which a plurality of areas and a plurality of sub-areas are filled in a distinguishable manner can be generated based on the second type of learning data.

  In the above aspect, a new image is input to one or more image recognition models, and is included in any of a plurality of regions included in the new image and a plurality of regions based on outputs of the one or more image recognition models. An image recognition unit that outputs a plurality of sub-regions and a plurality of regions and an area type for the plurality of sub-regions, and inputs the image of the plurality of regions and the image of the plurality of sub-regions to a character recognition model, and outputs the character recognition model A character recognition unit that outputs a character included in the plurality of regions and a character included in the plurality of sub-regions, and a correction unit that selects a correction rule according to the type of the region and corrects the character. May be provided.

  According to this aspect, even when the number of items included in the document is larger than the number of regions that can be distinguished by the image recognition model, all the regions can be identified stepwise by the image recognition unit. . In addition, the character recognition unit recognizes characters included in the plurality of regions and the plurality of sub-regions, and corrects characters according to the type of the region using the correction unit. Characters can be output with high precision.

  In the above aspect, the correction unit may correct the character by extracting a part of the character using one of the plurality of regular expressions.

  According to this aspect, when the character to be described is standardized, a part of the character can be extracted. For example, necessary numerical values can be extracted from characters.

  In the above aspect, the correction unit may correct the character by using the edit distance between the character and the plurality of candidate character strings to replace the character with one of the plurality of candidate character strings. .

  According to this aspect, when the characters described in the item are limited, a character string that cannot be described can be excluded and corrected to one of a plurality of candidate character strings.

  In the above aspect, the correction unit may correct the character by limiting the range of the character code.

  According to this aspect, when the range of the character code described in the item is limited, the character code that cannot be described can be excluded, and the character can be corrected by limiting the range of the character code.

  In the above aspect, the character recognition unit selects a character recognition model having a high degree of credibility of output from among the plurality of character recognition models, and, based on an output of the selected character recognition model, a character included in the plurality of regions and a plurality of characters. Characters included in the sub area may be output.

  According to this aspect, the character recognition accuracy can be further improved by selecting a character recognition model having a high output reliability.

  In the above aspect, the character recognition unit may select a character recognition model for each of the plurality of regions and the plurality of sub-regions.

  According to this aspect, a suitable character recognition model can be selected for each of the plurality of regions and the plurality of sub-regions, and the character recognition accuracy can be further improved.

  In the above aspect, the image recognizing unit selects an image recognition model having a high degree of credibility of output from one or a plurality of image recognition models, and, based on an output of the selected image recognition model, a plurality of regions and a plurality of sub-regions. And the type of area.

  According to this aspect, it is possible to further improve the recognition accuracy of a plurality of items included in the document by selecting an image recognition model having a high output credibility.

  In the above aspect, the image recognition unit may select an image recognition model for each type of document.

  According to this aspect, a suitable image recognition model can be selected for each type of document, and the recognition accuracy of a plurality of items included in the document can be further improved.

  In the above aspect, based on the input, at least one of the plurality of regions output by the image recognition unit, the plurality of sub-regions and the type of the region, the character output by the character recognition unit, and the character corrected by the correction unit. A correction unit that corrects any of them and adds the corrected data to the learning data may be further provided.

  According to this aspect, when either of the output of the image recognition model and the output of the character recognition model is erroneous, the data in which the error is corrected can be added to the learning data. It is possible to generate learning data that further improves accuracy.

  In the above aspect, a learning unit that re-learns at least one of one or a plurality of image recognition models and a plurality of character recognition models using a learning program in which predetermined parameters are set and learning data may be further provided. Good.

  According to this aspect, the output accuracy of the image recognition model and the character recognition model can be continuously improved by re-learning either the image recognition model or the character recognition model.

  In the above aspect, the learning unit changes a predetermined parameter when the output accuracy after re-learning of at least one of the one or more image recognition models and the character recognition models is lower than the output accuracy before re-learning, Re-learning of at least one of one or more image recognition models and character recognition models may be performed again.

  According to this aspect, if the output accuracy is reduced due to re-learning of either the image recognition model or the character recognition model, the parameters of the learning program are changed and re-learning is performed again, and the image recognition model and the character recognition model are re-executed. Output accuracy can be improved.

  A character recognition method according to another aspect of the present invention is a character recognition method for recognizing characters described in a document, wherein a plurality of areas are set for an image of the document based on an input, and Setting a plurality of sub-regions included in any of the plurality of regions; setting a region type for the plurality of regions and the plurality of sub-regions based on the input; Generating learning data used for supervised learning of one or a plurality of image recognition models including the position of the region, the positions of the plurality of sub-regions in the plurality of regions, and the type of the region.

  According to this aspect, even when the number of items included in the document is larger than the number of areas that can be distinguished by the image recognition model, the number of the plurality of areas is reduced by the number of areas that can be distinguished by the image recognition model. It is possible to set a smaller number, set a plurality of sub-areas included in any of the plurality of areas, and set an area corresponding to a plurality of items. This makes it possible to identify a plurality of areas and a plurality of sub-areas in a stepwise manner and to construct an image recognition model for identifying the type of those areas. Even if the document includes a plurality of items, a plurality of areas can be identified. An image recognition model can be constructed so as to appropriately extract a region of an item.

  According to another aspect of the present invention, there is provided a character recognition program for a processor provided in a character recognition device for recognizing a character described in a document, the first setting for setting a plurality of regions in an image of the document based on an input. A second setting unit that sets a plurality of sub-regions included in any of the plurality of regions based on the input, and sets a region type for the plurality of regions and the plurality of sub-regions based on the input The third setting unit generates learning data used for supervised learning of one or a plurality of image recognition models, including positions of a plurality of regions in the image, positions of a plurality of subregions in the plurality of regions, and types of the regions. Function as a generating unit.

  According to this aspect, even when the number of items included in the document is larger than the number of areas that can be distinguished by the image recognition model, the number of the plurality of areas is reduced by the number of areas that can be distinguished by the image recognition model. It is possible to set a smaller number, set a plurality of sub-areas included in any of the plurality of areas, and set an area corresponding to a plurality of items. This makes it possible to identify a plurality of areas and a plurality of sub-areas in a stepwise manner and to construct an image recognition model for identifying the type of those areas. Even if the document includes a plurality of items, a plurality of areas can be identified. An image recognition model can be constructed so as to appropriately extract a region of an item.

  According to the present invention, a character recognition device, a character recognition method, and a character recognition program capable of constructing an image recognition model so as to appropriately extract regions of a plurality of items even in a document including a plurality of items Is provided.

It is a figure showing the outline of the character recognition system concerning the embodiment of the present invention. It is a figure showing the functional block of the character recognition device concerning this embodiment. It is a figure showing the physical composition of the character recognition device concerning this embodiment. It is a figure showing an example of the setting screen of the character recognition device concerning this embodiment. It is a figure showing an example of the 1st kind of data for learning generated by the character recognition device concerning this embodiment. It is a figure showing an example of the 2nd type of data for learning generated by the character recognition device concerning this embodiment. 5 is a flowchart of a learning data generation process performed by the character recognition device according to the embodiment. 5 is a flowchart of a character recognition process performed by the character recognition device according to the embodiment. 5 is a flowchart of a character correction process performed by the character recognition device according to the embodiment. It is a flowchart of the relearning process performed by the character recognition device according to the present embodiment.

  Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, referred to as “the present embodiment”) will be described with reference to the drawings. In each of the drawings, the components denoted by the same reference numerals have the same or similar configurations.

  FIG. 1 is a diagram showing an outline of a character recognition system 100 according to an embodiment of the present invention. The character recognition system 100 according to the present embodiment includes a character recognition device 10, a user terminal 20, a character recognition model 30, a document image database DB1, and a master database DB2. The character recognition device 10, the user terminal 20, the character recognition model 30, the document image database DB1, and the master database DB2 may be able to communicate via a communication network N such as the Internet or a LAN (Local Area Network). The character recognition system 100 recognizes the characters described in the image of the document by the character recognition device 10, and when the result includes an error, accepts a correction from the user terminal 20 and continuously improves the recognition accuracy. It is a system that lets you.

  The document image database DB1 is a database that stores images of documents. The character recognition device 10 acquires an image of a document stored in the document image database DB1, receives an input specifying a plurality of items included in the document from the user terminal 20, and generates learning data.

  The master database DB2 is a database that stores master data of contents described in documents. For example, when the name of a branch of a bank is described in the document, the master database DB2 may store a list of currently existing branch names. In that case, the character recognition device 10 may acquire the master data from the master database DB2, and may match the recognized character with the master data.

  The character recognition model 30 is a learning model for recognizing characters included in an image, and is used by the character recognition device 10. The character recognition model 30 may be configured by a known learning model, and may be a model that recognizes handwritten characters and printed characters. In this example, the character recognition model 30 includes a first character recognition model 31 and a second character recognition model 32, but may include three or more models. The storage locations of the first character recognition model 31 and the second character recognition model 32 are arbitrary as long as they can be used via the communication network N. Note that the character recognition model 30 may be included in the character recognition device 10.

  FIG. 2 is a diagram illustrating functional blocks of the character recognition device 10 according to the present embodiment. The character recognition device 10 includes a learning processing unit 10L and a recognition processing unit 10R. The learning processing unit 10L includes a setting unit 11, a generation unit 12, a learning unit 13, learning data 15a, correction data 15b, and a correction unit 16. The recognition processing unit 10R includes an image recognition unit 14, a character recognition unit 17, a correction unit 18, dictionary data 19a, and master data 19b.

  The setting unit 11 includes a first setting unit 11a, a second setting unit 11b, and a third setting unit 11c. The setting unit 11 acquires an image of a document from the document image database DB1 and receives an input from the user terminal 20. The first setting unit 11a sets a plurality of areas for an image of a document based on the input. Here, the plurality of areas may be set so as to surround a plurality of items included in the document. The first setting unit 11a may set an area including one item, or may set an area including a plurality of items.

  The second setting unit 11b sets a plurality of sub-regions included in any of the plurality of regions based on the input. The plurality of sub-regions may be set inside the region set by the first setting unit 11a. The second setting unit 11b may set a sub area including one item, or may set a sub area including a plurality of items. The second setting unit 11b may set a plurality of sub-regions included in any of the plurality of sub-regions based on the input. That is, a plurality of sub-regions may be set inside the region by the second setting unit 11b, and a plurality of sub-regions may be set further inside the sub-region.

  The third setting unit 11c sets an area type for a plurality of areas and a plurality of sub-areas based on the input. One type of region may be set for each of the plurality of regions and the plurality of sub-regions, but one or more types of regions may be set for the plurality of regions and the plurality of sub-regions. The type of area may represent the content of an item corresponding to the area. For example, in the case of a document relating to a bank account, the type of area may be an account number or a branch name.

  The generation unit 12 generates learning data used for supervised learning of one or a plurality of image recognition models, including positions of a plurality of regions in a document image, positions of a plurality of subregions in a plurality of regions, and types of regions. I do. The generation unit 12 determines the positions of the plurality of regions set by the first setting unit 11a, the positions of the plurality of sub-regions set by the second setting unit 11b, and the third setting unit 11c with respect to the image of the document. The learning data may be generated by associating the plurality of regions and the types of the plurality of sub-regions set by the above. The learning data generated by the generation unit 12 is stored as learning data 15a.

  According to the character recognition device 10 according to the present embodiment, even when the number of items included in the document is larger than the number of regions that can be distinguished by the image recognition model, the number of the plurality of regions is determined by the image recognition model. , The number of areas that can be distinguished is set smaller than the number of areas that can be distinguished, a plurality of sub-areas included in any of the plurality of areas are set, and areas corresponding to a plurality of items can be set. This makes it possible to identify a plurality of areas and a plurality of sub-areas in a stepwise manner and to construct an image recognition model for identifying the type of those areas. Even if the document includes a plurality of items, a plurality of areas can be identified. An image recognition model can be constructed so as to appropriately extract a region of an item.

  The generation unit 12 may generate a plurality of types of learning data used for supervised learning of one or a plurality of image recognition models. By generating a plurality of types of learning data, a plurality of regions, a plurality of sub-regions, and a type of region are set once, so that a plurality of types of learning data used for supervised learning of one or a plurality of image recognition models. Can be generated together, and a plurality of types of learning data can be efficiently generated.

  The generation unit 12 generates a first type of learning data in which the outlines of the plurality of regions and the plurality of sub-regions are represented in different modes for each type of region, and the plurality of regions and the plurality of sub-regions for each type of region. And the second type of learning data filled in a different manner. Examples of the first type of learning data and the second type of learning data will be described in detail with reference to FIGS. With the first type of learning data, it is possible to generate a first image recognition model 14a that overwrites an image with a bounding box surrounding a plurality of regions and a plurality of sub-regions in a distinguishable manner. Further, the second type of learning data can be used to generate a second image recognition model 14b that outputs an image in which a plurality of areas and a plurality of sub-areas are filled in a distinguishable manner.

  The learning unit 13 performs supervised learning of one or a plurality of image recognition models using the learning program 13a in which predetermined parameters are set and the learning data 15a. In the case of this example, the learning unit 13 uses the first learning program and the first type of learning data to overwrite an image with a bounding box that surrounds the plurality of regions and the plurality of sub-regions in a distinguishable manner. The image recognition model 14a may be generated. The first image recognition model 14a may be configured by a CNN (Convolutional Neural Network), and more specifically, may be configured by an SSD (Single Shot Multibox Detector), Faster R-CNN, or Retina Net. Further, the learning unit 13 uses the second learning program and the second type of learning data to generate a second image recognition model 14b that outputs an image in which a plurality of regions and a plurality of sub-regions are filled in a distinguishable manner. May be generated. The second image recognition model 14b may be configured by a GAN (Generative Adversarial Network), and more specifically, may be configured by a VAE (Variational Autoencoder).

  The image recognition unit 14 inputs a new image IMG to one or a plurality of image recognition models, and based on an output of the one or a plurality of image recognition models, selects one of a plurality of regions and a plurality of regions included in the new image IMG. It outputs the plurality of sub-regions included in the, and the plurality of regions and the type of region for the plurality of sub-regions. Here, the new image IMG is an image of a new document. In this example, the image recognizing unit 14 uses one of the first image recognition model 14a and the second image recognition model 14b generated by the learning unit 13 to specify the area of the image corresponding to the item of the document and the type of the area. Is output. Even if the number of items included in the document is larger than the number of regions that can be distinguished by the image recognition model, the image recognition unit 14 can identify all the regions in a stepwise manner.

  The character recognition unit 17 inputs the images of the plurality of regions and the images of the plurality of sub-regions to the character recognition model 30, and based on the output of the character recognition model 30, outputs the characters included in the plurality of regions and the plurality of sub-regions. Outputs the characters included. In this example, the character recognizing unit 17 outputs a character included in the image using one of the first character recognition model 31 and the second character recognition model 32. The character recognition unit 17 can recognize characters included in a plurality of areas and a plurality of sub-areas.

  The correction unit 18 selects a correction rule according to the type of the area, and corrects a character. The correction unit 18 may correct the character on a rule basis according to a predetermined correction rule. The correction unit 18 corrects characters according to the type of area, so that characters described in a plurality of items included in the document can be output with high accuracy.

  The correction unit 18 may correct a character by extracting a part of the character using one of a plurality of regular expressions. Thereby, when the character to be described is standardized, a part of the character can be extracted. For example, necessary numerical values can be extracted from characters. Here, the correction unit 18 may select a regular expression to be used for correction with reference to the dictionary data 19a.

  The correction unit 18 uses the edit distance between the character recognized by the character recognition unit 17 and the plurality of candidate character strings to convert the character recognized by the character recognition unit 17 into a plurality of candidate character strings. The character may be corrected by replacing the character. Thus, when the characters described in the item are limited, the character string that cannot be described can be excluded and corrected to one of the plurality of candidate character strings. For example, in the case of an item in which a bank branch name is described, a plurality of candidate character strings can be used as the currently existing branch name, and the described characters can be corrected to any of the currently existing branch names. Here, the correction unit 18 may select a plurality of candidate character strings with reference to the master data 19b. The master data 19b may acquire the latest data from the master database DB2 when the master database DB2 is updated.

  The correction unit 18 may correct the character by limiting the range of the character code. Thus, when the range of character codes described in the item is limited, character codes that cannot be described can be excluded, and characters can be corrected by limiting the range of character codes. For example, when the range of the character code described in the item is limited to the range of the character code representing a numerical value, a correction such as replacing the alphabet “O” with the number “0 (zero)” is performed. be able to.

  The character recognition unit 17 selects a character recognition model having a high degree of credibility of output from the plurality of character recognition models, and based on an output of the selected character recognition model, determines a character included in the plurality of regions and a plurality of sub-regions. The characters included may be output. In the case of the present example, the character recognition unit 17 selects a character recognition model having a high degree of credibility of output from the first character recognition model 31 and the second character recognition model 32, and based on the output of the selected character recognition model. Characters included in a plurality of areas and characters included in a plurality of sub-areas may be output. As described above, by selecting a character recognition model having high output reliability, the character recognition accuracy can be further improved.

  The character recognition unit 17 may select a character recognition model for each of the plurality of regions and the plurality of sub-regions. For example, the first character recognition model 31 may be selected for an area where an account number is described, and the second character recognition model 32 may be selected for an area where a branch name is described. In this way, a suitable character recognition model can be selected for each of the plurality of regions and the plurality of sub-regions, and the character recognition accuracy can be further improved.

  The image recognition unit 14 selects an image recognition model having a high degree of credibility of output from one or a plurality of image recognition models, and generates a plurality of regions, a plurality of sub-regions, and a region based on the output of the selected image recognition model. The type may be output. In the case of this example, the image recognition unit 14 selects an image recognition model having a high degree of credibility of output from the first image recognition model 14a and the second image recognition model 14b, and based on the output of the selected image recognition model. A plurality of areas, a plurality of sub-areas, and a type of area may be output. As described above, by selecting an image recognition model having a high output credibility, the recognition accuracy of a plurality of items included in a document can be further improved.

  The image recognition unit 14 may select an image recognition model for each type of document. For example, the first image recognition model 14a may be selected for documents relating to bank accounts, and the second character recognition model 32 may be selected for documents relating to vehicle inspection (automobile inspection registration system). In this way, an appropriate image recognition model can be selected for each type of document, and the recognition accuracy of a plurality of items included in the document can be further improved.

  The correcting unit 16 corrects the plurality of regions, the plurality of sub-regions and the types of the regions output by the image recognizing unit 14, the characters output by the character recognizing unit 17, and the correction unit 18 based on the input. At least one of the characters is corrected, and the corrected data is added to the learning data 15a. The correction unit 16 is configured to output at least one of the plurality of regions, the plurality of sub-regions, and the types of the regions output by the image recognition unit 14, the character output by the character recognition unit 17, and the character corrected by the correction unit 18. The correction data 15b obtained by correcting the above may be stored, and the correction data 15b may be periodically added to the learning data 15a. Further, the correction unit 16 may add the correction data 15b to the dictionary data 19a.

  When the output of either the image recognition model or the character recognition model is erroneous, the correction unit 16 can add the data in which the error has been corrected to the learning data, and can improve the output accuracy of the image recognition model and the character recognition model. Can be generated for learning.

  The learning unit 13 may re-learn at least one of one or a plurality of image recognition models and a plurality of character recognition models using the learning program 13a and the learning data 15a. For example, the learning unit 13 may re-learn one or a plurality of image recognition models when the correction unit 16 adds the learning data 15a. By performing re-learning of either the image recognition model or the character recognition model, the output accuracy of the image recognition model and the character recognition model can be continuously improved.

  The learning unit 13 changes predetermined parameters of the learning program 13a when the output accuracy after re-learning of at least one of the one or more image recognition models and the character recognition models is lower than the output accuracy before re-learning. The re-learning of at least one of the image recognition model and the character recognition model may be performed again. Here, the predetermined parameter may be a hyperparameter of a stochastic gradient descent method such as a learning rate. Thereby, if the output accuracy is reduced due to re-learning of either the image recognition model or the character recognition model, the parameters of the learning program are changed and re-learning is performed again, and the output accuracy of the image recognition model and the character recognition model is reduced. Can be improved.

  FIG. 3 is a diagram illustrating a physical configuration of the character recognition device 10 according to the present embodiment. The character recognition device 10 includes a CPU (Central Processing Unit) 10a corresponding to an operation unit, a RAM (Random Access Memory) 10b corresponding to a storage unit, a ROM (Read only Memory) 10c corresponding to a storage unit, and a communication unit. 10d, an input unit 10e, and a display unit 10f. These components are connected to each other via a bus so that data can be transmitted and received. In this example, a case will be described in which the character recognition device 10 is configured by one computer, but the character recognition device 10 may be realized by combining a plurality of computers. In addition, the configuration illustrated in FIG. 3 is an example, and the character recognition device 10 may have other configurations, or may not have some of these configurations.

  The CPU 10a is a control unit that performs control related to execution of a program stored in the RAM 10b or the ROM 10c and calculates and processes data. The CPU 10a is a calculation unit that executes a program (character recognition program) for recognizing characters described in an image of a document. The CPU 10a receives various data from the input unit 10e and the communication unit 10d, and displays a calculation result of the data on the display unit 10f and stores the calculation result in the RAM 10b and the ROM 10c.

  The RAM 10b is a storage unit in which data can be rewritten, and may be composed of, for example, a semiconductor storage element. The RAM 10b may store data such as a character recognition program executed by the CPU 10a and images of documents. These are merely examples, and the RAM 10b may store data other than these or some of them may not be stored.

  The ROM 10c is a storage unit from which data can be read, and may be configured by, for example, a semiconductor storage element. The ROM 10c may store, for example, a character recognition program or data that is not rewritten.

  The communication unit 10d is an interface that connects the character recognition device 10 to another device. The communication unit 10d may be connected to a communication network N such as the Internet.

  The input unit 10e accepts data input from a user, and may include, for example, a keyboard and a touch panel.

  The display unit 10f is for visually displaying the calculation result by the CPU 10a, and may be configured by, for example, an LCD (Liquid Crystal Display). The display unit 10f may display an image of a document to be processed or a character recognition result.

  The character recognition program may be provided by being stored in a computer-readable storage medium such as the RAM 10b or the ROM 10c, or may be provided via the communication network N connected by the communication unit 10d. In the character recognition device 10, the various operations described with reference to FIG. 2 are realized by the CPU 10a executing the character recognition program. Note that these physical configurations are merely examples, and are not necessarily independent configurations. For example, the character recognition device 10 may include an LSI (Large-Scale Integration) in which the CPU 10a and the RAM 10b or the ROM 10c are integrated.

  FIG. 4 is a diagram illustrating an example of the setting screen DP of the character recognition device 10 according to the present embodiment. The setting screen DP may be a screen displayed on the user terminal 20, and includes a document image IMG1, a pointer PT, a setting mode selection S1, and an area setting S2. The user selects one of the radio buttons of the setting mode selection S1 and one of the radio buttons of the area setting S2 with the pointer PT, and sets a plurality of areas and a plurality of sub-areas for the image IMG1.

  In the area setting S2, a radio button for setting an area type “1. red”, a radio button for setting an area type “2. blue”, and an area type “3. yellow” are set. A radio button and a radio button for setting an area type of “4. pink” are included. That is, in this example, four types of regions can be set. The setting mode selection S1 includes a “base_image” radio button for setting a plurality of areas, and a “crop_red” radio button for setting a sub-area for a “1.red” type area. , A radio button of “crop_blue” for setting a sub-area for an area of “2. blue” type, and a radio of “crop_yellow” for setting a sub-area for an area of type “3. yellow” A button and a radio button of “crop_pink”, which is a mode for setting a sub-area for an area of “4. pink” type, are included.

  In this example, the image IMG1 is represented by a plurality of regions including a first region R1 indicated by a solid line, a second region R2 indicated by a broken line, a third region R3 indicated by a dashed line, and a two-dot chain line. The indicated fourth region R4 is set. Here, the line type corresponds to the type of the area selected by the area setting S2. In this example, the solid line corresponds to “1.red”, the dashed line corresponds to “2.blue”, the one-dot chain line corresponds to “3.yellow”, and the two-dot chain line corresponds to “4.pink”. .

  The first region R1 includes a first sub-region R1a indicated by a solid line and a second sub-region R1b indicated by a broken line. In this way, by setting a plurality of sub-regions inside the region, even if only four types of regions can be recognized by the image recognition model, learning data that distinguishes five or more regions is generated, It is possible to identify five or more regions stepwise by the image recognition model.

  In this example, “ABC” is described in the first sub-region R1a, “abc” is described in the second sub-region R1b, “DEF” is described in the second region R2, and “1234” is described in the third region R3. And "G-56" is described in the fourth region R4. The character recognition device 10 recognizes the first sub-region R1a, the second sub-region R1b, the second region R2, the third region R3, and the fourth region R4 by the image recognition unit 14, and the character recognition unit 17 Recognize the characters described in the area. Further, the character recognition device 10 corrects the character recognized by the character recognition unit 17 by the correction unit 18. For example, when it is determined in advance that a two-digit numerical value is described in the fourth region R4 after the character string “G-”, the correction unit 18 uses a regular expression to write the character “G-56”. The character may be corrected by extracting “56” from among them. Further, for example, when it is predetermined that only a numerical value is described in the third region R3, the correction unit 18 limits the range of the character code, and changes, for example, “l” to “1 (one)”. And the character “1234” may be corrected.

  FIG. 5 is a diagram illustrating an example of the first type of learning data D1 generated by the character recognition device 10 according to the present embodiment. The first type of learning data D1 is data used for supervised learning of the first image recognition model 14a that overwrites an image with a bounding box surrounding a plurality of regions and a plurality of sub-regions in a distinguishable manner.

  The first type of learning data D1 includes a first area R1 indicated by a solid line, a second area R2 indicated by a broken line, a third area R3 indicated by a dashed line, and a fourth area indicated by a two-dot chain line. The first region R1 includes a first sub-region R1a indicated by a solid line and a second sub-region R1b indicated by a broken line. The generation unit 12 may generate the first type of learning data D1 by directly using the outlines of the plurality of regions and the plurality of sub-regions input by the user terminal 20, or may input the learning data D1 by the user terminal 20. Data augmentation may be performed by slightly translating the contours of the plurality of regions and the plurality of sub-regions in a plurality of directions to generate the first type of learning data D1.

  FIG. 6 is a diagram illustrating an example of the second type of learning data D2 generated by the character recognition device 10 according to the present embodiment. The second type of learning data D2 is data used for supervised learning of the second image recognition model 14b that outputs an image in which a plurality of areas and a plurality of sub-areas are filled in a distinguishable manner.

  The second type of learning data D2 is indicated by a first solid area F1 indicated by a solid line, a second solid area F2 indicated by a broken line, a third solid area F3 indicated by a dashed line, and a two-dot chain line. The first filled area F1 includes a first sub-filled area F1a shown by a solid line and a second sub-filled area F1b shown by a broken line. The plurality of filled regions and the plurality of sub-filled regions may be filled with a color corresponding to the type of the region. In this example, the difference in color is represented by the difference in hatching. The generation unit 12 may generate the second type of learning data D2 by filling the insides of the outlines of the plurality of regions and the plurality of sub-regions input by the user terminal 20. Data augmentation is performed by slightly translating the contours of the plurality of input regions and the plurality of sub-regions in a plurality of directions, and the second type of learning data D2 is obtained by filling the insides of the contours. May be generated.

  FIG. 7 is a flowchart of a learning data generation process performed by the character recognition device 10 according to the present embodiment. First, the character recognition device 10 sets a plurality of regions for a document image based on the input (S10). Further, the character recognition device 10 sets a plurality of sub-regions included in any of the plurality of regions based on the input (S11). Further, the character recognition device 10 sets the type of the area for the plurality of areas and the plurality of sub areas based on the input (S12). Note that these inputs may be input from the user terminal 20.

  After that, the character recognition device 10 generates the first type of learning data (S13), and generates the second type of learning data (S14). Although the case where two types of learning data are generated has been described in the present example, the character recognition device 10 generates a plurality of types of learning data corresponding to the number of the plurality of image recognition models used by the image recognition unit 14. May be generated. Thus, the learning data generation processing ends.

  FIG. 8 is a flowchart of a character recognition process performed by the character recognition device 10 according to the present embodiment. First, the character recognition device 10 selects an image recognition model based on the type of the document (S20). Then, the character recognition device 10 inputs the new image to the selected image recognition model, and outputs a plurality of regions, a plurality of sub-regions, and a type of region (S21).

  Next, the character recognition device 10 selects a character recognition model for each of the plurality of regions and the plurality of sub-regions (S22). Then, the character recognition device 10 inputs the images of the plurality of regions and the images of the plurality of sub-regions to the selected character recognition model, and outputs characters included in the plurality of regions and characters included in the plurality of sub-regions. (S23).

  Thereafter, the character recognition device 10 corrects the character according to the type of the area (S24). The character correction process will be described in detail with reference to the following diagram. Thereafter, the character recognition device 10 outputs the read character (S25). Thus, the character recognition processing ends.

  FIG. 9 is a flowchart of a character correction process performed by the character recognition device 10 according to the present embodiment. FIG. 14 shows an example of the details of the character correcting process (S24) in FIG.

  When the type of the region corresponds to the extraction of a numerical value (S241: YES), the character recognition device 10 corrects the character by extracting a part of the character using one of a plurality of regular expressions. (S242). In this example, a case where a numerical value is extracted is shown. However, when extracting a part of a character by using a regular expression, a character other than a numerical value such as an alphabetic character or a kanji may be extracted, or a numerical value, an alphabetic character, and a kanji character may be extracted. May be extracted.

  If the type of the area does not correspond to the extraction of the numerical value (S241: NO) but corresponds to the master match (S243: YES), the character recognition device 10 causes the character recognition unit 17 to recognize the character. The character is corrected by replacing the character recognized by the character recognizing unit 17 with one of the plurality of candidate character strings using the editing distance between the character string and the candidate character strings (S244). .

  When the type of the area does not correspond to the extraction of the numerical value (S241: NO), does not correspond to the master match (S243: NO), and the character code range is limited to only the numerical value (S245). : YES), the character code is corrected by limiting the range of the character code to the range representing the numerical value (S246). Note that the case divisions shown in this example are exemplifications, and the order of each case division is arbitrary, and different case divisions may be included. Thus, the character correction processing ends.

  FIG. 10 is a flowchart of the relearning process executed by the character recognition device 10 according to the present embodiment. The re-learning process may be performed when there is an error in the recognition result after the character recognition process is performed or when the master database DB2 is updated.

  First, the character recognition device 10 receives an input of correction data from the user terminal 20 (S30). Then, the character recognition device 10 corrects the plurality of output regions, the plurality of sub-regions, and the type of the region based on the input from the user terminal 20 (S31). Further, the character recognition device 10 corrects the output character and the corrected character based on the input from the user terminal 20 (S32).

  Thereafter, the character recognition device 10 adds the data corrected for the plurality of regions, the plurality of sub-regions, and the type of the region to the learning data (S33), and stores the corrected data for the characters and the corrected characters in the dictionary data. (S34).

  When the master database is updated (S35: YES), the character recognition device 10 acquires the latest data and updates the master data 19b (S36).

  Thereafter, the character recognition device 10 executes a re-learning process of the image recognition model and the character recognition model (S37). When the output accuracy of the image recognition model and the character recognition model is reduced as a result of the re-learning process (S38: YES), the character recognition device 10 changes the parameters of the learning program 13a (S39), and executes the image recognition model and the character recognition. A model re-learning process is executed (S37). On the other hand, when the output accuracy of the image recognition model and the character recognition model is improved as a result of the relearning process (S38: NO), the image recognition model and the character recognition model are updated (S40). Thus, the re-learning process ends. If the output accuracy of the image recognition model and the character recognition model does not improve even if the parameters are changed, the character recognition device 10 does not update the image recognition model and the character recognition model, but performs the re-learning process with the previous version. May be terminated.

  The embodiments described above are intended to facilitate understanding of the present invention, and are not intended to limit and interpret the present invention. The components included in the embodiment and their arrangement, material, condition, shape, size, and the like are not limited to those illustrated, but can be appropriately changed. It is also possible to partially replace or combine the configurations shown in the different embodiments.

  Reference numeral 10: character recognition device, 10a: CPU, 10b: RAM, 10c: ROM, 10d: communication unit, 10e: input unit, 10f: display unit, 10L: learning processing unit, 10R: recognition processing unit, 11: setting unit, 11a: first setting unit, 11b: second setting unit, 11c: third setting unit, 12: generation unit, 13: learning unit, 13a: learning program, 14: image recognition unit, 14a: first image recognition model, 14b: second image recognition model, 15a: learning data, 15b: correction data, 16: correction unit, 17: character recognition unit, 18: correction unit, 19a: dictionary data, 19b: master data, 20: user terminal, Reference numeral 30: character recognition model, 31: first character recognition model, 32: second character recognition model, DB1: document image database, DB2: master database, 100: character recognition system

Claims (16)

A character recognition device for recognizing characters described in a document,
A first setting unit that sets a plurality of regions for the image of the document based on the input;
A second setting unit configured to set a plurality of sub-regions included in any of the plurality of regions based on an input;
A third setting unit configured to set an area type for the plurality of areas and the plurality of sub-areas based on the input;
Generating learning data for use in supervised learning of one or more image recognition models, including positions of the plurality of regions in the image, positions of the plurality of sub-regions in the plurality of regions, and types of the regions; Department and
A character recognition device comprising: The generation unit generates a plurality of types of learning data used for supervised learning of the one or more image recognition models.
The character recognition device according to claim 1. The generation unit may include a first type of learning data in which the outlines of the plurality of regions and the plurality of sub-regions are expressed in different modes for each type of the region, and the plurality of regions and the plurality of sub-regions. A second type of learning data that is filled in a different manner for each type of region.
The character recognition device according to claim 2. A new image is input to the one or more image recognition models, and based on an output of the one or more image recognition models, a plurality of regions included in the new image is included in any of the plurality of regions. An image recognition unit that outputs a plurality of sub-regions and a type of a region for the plurality of regions and the plurality of sub-regions,
The image of the plurality of regions and the image of the plurality of sub-regions are input to a character recognition model, and based on the output of the character recognition model, the characters included in the plurality of regions and the characters included in the plurality of sub-regions A character recognition unit that outputs
A correction unit that selects a correction rule according to the type of the area, and corrects the character;
The character recognition device according to any one of claims 1 to 3, further comprising: The correction unit corrects the character by extracting a part of the character using one of a plurality of regular expressions,
The character recognition device according to claim 4. The correction unit corrects the character by using the edit distance between the character and a plurality of candidate character strings to replace the character with one of the plurality of candidate character strings.
The character recognition device according to claim 4. The correction unit limits the range of a character code, and corrects the character.
The character recognition device according to claim 4. The character recognition unit selects a character recognition model having a high degree of credibility of output among a plurality of character recognition models, and, based on an output of the selected character recognition model, a character included in the plurality of regions and the plurality of characters. Output the characters contained in the sub-area,
The character recognition device according to claim 4. The character recognition unit selects a character recognition model for each of the plurality of regions and the plurality of sub-regions,
The character recognition device according to claim 8. The image recognizing unit selects an image recognition model having a high degree of credibility of output from the one or more image recognition models, and, based on the output of the selected image recognition model, the plurality of regions and the plurality of sub-regions. Outputting the area and the type of the area,
The character recognition device according to claim 4. The image recognition unit selects the image recognition model for each type of the document,
The character recognition device according to claim 10. Based on the input, the plurality of regions output by the image recognition unit, the plurality of sub-regions and the type of the region, the character output by the character recognition unit, and the character corrected by the correction unit Correcting at least one of the, further comprising a correction unit to add the corrected data to the learning data,
The character recognition device according to claim 4. A learning unit configured to re-learn at least one of the one or more image recognition models and the plurality of character recognition models using a learning program in which predetermined parameters are set and the learning data;
The character recognition device according to claim 4. The learning unit, when the output accuracy after re-learning of at least one of the one or more image recognition models and the character recognition model is lower than the output accuracy before re-learning, changes the predetermined parameter, Re-executing at least one of the one or more image recognition models and the character recognition model;
The character recognition device according to claim 13. A character recognition method for recognizing characters described in a document,
Based on the input, setting a plurality of areas for the image of the document;
Based on the input, setting a plurality of sub-regions included in any of the plurality of regions,
Based on the input, for the plurality of regions and the plurality of sub-regions, to set the type of region,
Generating learning data for use in supervised learning of one or more image recognition models, including positions of the plurality of regions in the image, positions of the plurality of sub-regions in the plurality of regions, and types of the regions. When,
Character recognition method including. A processor provided in a character recognition device that recognizes characters described in documents,
A first setting unit that sets a plurality of regions for the image of the document based on the input;
A second setting unit configured to set a plurality of sub-regions included in any of the plurality of regions based on the input;
A third setting unit that sets an area type for the plurality of areas and the plurality of sub-areas based on the input; and a position of the plurality of areas in the image, the plurality of sub-areas in the plurality of areas. A generating unit that generates learning data used for supervised learning of one or more image recognition models, including a position of a region and a type of the region,
Character recognition program to function as.

Images