JP2019079347A - Character estimation system, character estimation method, and character estimation program - Google Patents

Abstract

To enable accurate extraction of text information in various documents.SOLUTION: A document image acquisition unit 21 acquires a document image. A character type estimation unit 22 determines position/size information of one or more text regions in the document image and a character type of a text in each of the one or more text regions. A text identification unit 23 executes character recognition processing for the estimated character type for the estimated text region to specify the text in the estimated text region as text data. The character type estimation unit 22 includes a machine-trained deep neural network, and input data to the deep neural network is the above-mentioned document image, and output data of the deep neural network is the above-mentioned position/size information and character type for each of the above-mentioned one or more text regions.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a character type estimation system, a character type estimation method, and a character type estimation program.

  In a form identification system, a form format table is created in advance by the user, and the form format table includes field information indicating the position, size, character type, and the like of the character recognition target area designated by the user. Then, based on the form format (ie, field information), character information (text data) in the form is acquired from the image data of the form image.

JP, 2016-48444, A

  However, in the above-described system, a form format table corresponding to the layout (format) of a document such as a form must be prepared in advance, and for a plurality of documents having different formats, the same number as the number of the documents It is necessary to create a form format table, and extracting character information in various documents is practically difficult in view of the time and effort of creating the form format table.

  The present invention has been made in view of the above problems, and an object of the present invention is to obtain a character type estimation system, a character type estimation method, and a character type estimation program that can accurately extract character information in various documents.

  A character type estimation system according to the present invention includes a document image acquisition unit for acquiring a document image, position / size information of one or more text regions in the document image, and a text region for each of the one or more text regions. A character type estimation unit for estimating the character type of the text in the text, and a text identification unit for identifying text in the estimated text area as text data by executing character recognition processing for the estimated character type for the estimated text area And The character type estimation unit is provided with a machine-trained deep neural network, and the input data of the deep neural network is the document image described above, and the output data of the deep neural network is one or more text regions described above The above-mentioned position / size information and character type for each of.

  A character type estimation method according to the present invention is a document image acquisition step of acquiring a document image, and the above-described machine-learned deep neural network, position / size information of one or more text regions in the document image, and The character type estimation step of estimating the character type of the text in the text region for each of the one or more text regions, and the character recognition process for the character type estimated for the estimated text region are performed, and within the estimated text region And a text identifying step of identifying the text of the text as text data.

  A character type estimation program according to the present invention causes a computer to function as the document image acquisition unit described above, the character type estimation unit described above, and the text specification unit described above.

  According to the present invention, it is possible to obtain a character type estimation system, a character type estimation method, and a character type estimation program that enable accurate extraction of character information in various documents.

  The above or other objects, features and advantages of the present invention will become more apparent from the following detailed description in conjunction with the accompanying drawings.

FIG. 1 is a block diagram showing the configuration of a character type estimation system according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a document image. FIG. 3 is a diagram for explaining a text area estimated for the document image shown in FIG. FIG. 4 is a flow chart for explaining the operation of the character type estimation system shown in FIG.

  Hereinafter, embodiments of the present invention will be described based on the drawings.

  FIG. 1 is a block diagram showing the configuration of a character type estimation system according to an embodiment of the present invention. The character type estimation system shown in FIG. 1 is configured by one information processing apparatus, but the processing units described later may be distributed to a plurality of information processing apparatuses capable of data communication with each other. In addition, such a plurality of information processing apparatuses may include a GPU (Graphics Processing Unit) that performs specific processing in parallel.

  The character type estimation system shown in FIG. 1 includes a storage device 1, a communication device 2, an image reading device 3, and an arithmetic processing device 4.

  The storage device 1 is a non-volatile storage device such as a flash memory or a hard disk, and stores various data and programs.

  Here, the character type estimation program 11 is stored in the storage device 1, and setting data (such as a coefficient setting value of a deep neural network described later) is stored as necessary. The character type estimation program 11 may be stored in a portable computer readable recording medium such as a CD (Compact Disk). In that case, for example, the program 11 is installed from the recording medium to the storage device 1. Also, the program 11 may be a single program or a collection of a plurality of programs.

  The communication device 2 is a device capable of data communication such as a network interface, a peripheral device interface, and a modem, and performs data communication with other devices as needed. The image reader 3 optically reads a document image from the document and generates image data (such as raster image data) of the document image.

  The arithmetic processing unit 4 is a computer provided with a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), etc., and loads a program from the ROM, the storage device 1 or the like to the RAM, and By being executed, it operates as various processing units.

  Here, by executing the program 11, the arithmetic processing unit 4 functions as the document image acquisition unit 21, the character type estimation unit 22, the text specification unit 23, the data file generation unit 24, the verification unit 25, and the machine learning processing unit 26. Operate.

  The document image acquisition unit 21 acquires a document image as image data. The document image is an image (raster image) such as a document (including a receipt), a bill, a document such as a delivery note, and other fixed documents in an organization such as a company. For example, the document image acquisition unit 21 reads a document image as image data stored in the storage device 1, acquires a document image as image data received by the communication device 2, or the image reading device 3 The document image as the generated image data is acquired.

  The character type estimation unit 22 estimates the character type of the text in each text region for each of position / size information of one or more text regions (rectangular regions) in the document image and the one or more text regions. Specifically, the character type estimation unit 22 is provided with a machine-learned deep neural network, and uses the deep neural network to estimate the position / size information of each text region and the character type of the text in the text region. The input data of this deep neural network is a document image (that is, the pixel values of the document image), and the output data of this deep neural network is the position / size information and character type for each of the one or more text regions described above. It is.

  The character type to be estimated is any of the character types used as an output value of training data (a pair of input value and output value) used at the time of machine learning of the deep neural network.

  For example, the character type indicates the feature (internal attribute) of the text itself (a) language type (Japanese, English, etc.), (b) writing system (hiragana, katakana, kanji, etc.), (c) number? It includes classifications such as (d) handwritten or printed characters, and (e) their combination.

  Also, the character type may indicate an external attribute of the text (that is, an attribute to be added to the text) in place of or in addition to the feature of the above-mentioned text itself. Well, for example, (f) monetary amount (number with a currency mark such as \), (g) table heading (such as header character), (h) email (combination of English text and @ mark with domain description text ) (I) URL (Uniform Resource Locator), (j) phone number, (k) address, (l) date, (m) seal impression (outer frame such as circle or rectangle and the text in the outer frame) Classification may be included.

  Furthermore, as the special character type, the character type is a classification of figures that do not contain text (for example, (a) barcode, (b) two-dimensional code, (c) logo mark, (d1) checked check box, ( d2) may include unchecked checkboxes and the like).

  Here, the deep neural network is, for example, a Boltzmann machine with a limited number of hidden layers or more. That is, it is a neural network machine-learning by deep learning. In addition, as the configuration and machine learning of this deep neural network, known ones can be used. For example, CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), RNN (Recursive Neural Network) or the like may be used for this deep neural network. Furthermore, a Long Short-Term Memory (LSTM) method, a Connectionist Temporal Classification (CTC) method, etc. may be applied to such a deep neural network, as necessary.

  Also, for machine learning of this deep neural network, raster image data (that is, bit map image data) (input data) of document images of document images of various documents and one or more text areas of the document image The position / size information of and the pair with the character type (output data) are used as training data.

  FIG. 2 is a diagram showing an example of a document image. FIG. 3 is a diagram for explaining a text area estimated for the document image shown in FIG.

  A document image 101 shown in FIG. 2 is an image of a delivery note. When the document image 101 is input to the deep neural network, the position / size information of the text area and the character type of the text area are output from the deep neural network as shown by the dashed line rectangle in FIG. The position / size information of the text area may be, for example, coordinate values of two points on a diagonal of the rectangular text area, or coordinates of representative points such as a center point of a rectangular text area, a specific vertex, etc. It may be vertical or horizontal.

  For example, as the text area 111 in the document image 101 shown in FIG. 3, the character type “Katakana” (that is, katakana in Japanese print) is estimated together with the position / size information thereof, and the position / size information as the text area 112. In addition, the character type "alphabet" (that is, the alphabet of the print character in English) is estimated, and the character type "number" (that is, the number of the print character) is estimated together with the position / size information as the text area 113. As 114, the character type "handwritten kanji" (that is, Japanese handwritten kanji) is estimated along with the position / size information.

  The text specifying unit 23 extracts a partial image of the text area from the document image based on the estimated position / size information, and recognizes the character type for the character type estimated for the estimated text area (that is, the extracted partial image) Processing is performed to identify the text in the estimated text area as text data (ie, character code).

  For example, in the document image 101 shown in FIG. 3, the character types of the text areas 111 to 114 are estimated as described above, and the text specifying unit 23 generates the text data of the text areas 111 to 114. For (partial image), character recognition processing for katakana in print is executed, and for the image (partial image) in text region 112, character recognition processing for alphabets in print is executed. Character recognition processing for printed numbers is performed for images (partial images), and character recognition processing for handwritten Japanese characters for images (partial images) in the text area 114 is performed.

  Note that character recognition processing of other character types is also possible even in character recognition processing for a specific character type. However, the character recognition rate may be lower or the recognition error rate may be higher than the particular character type.

  When the character type is a special character type not including the above-described text, the text specification unit 23 may execute processing corresponding to the special character type on the image of the text region. For example, if the character type is a bar code or a two-dimensional code, the text identification unit 23 decodes the bar code or the two-dimensional code and converts the bar code or the two-dimensional code into text data or binary data You may do so. Also, for example, when the character type is a logo mark, the text specification unit 23 may cut out the image of the text area from the document image and output it as a logo image. Also, for example, if the character type is a checked check box or an unchecked check box, the text identifying unit 23 may check the checked check box or the unchecked text as the text of the text area. A unique code (text data or binary data) corresponding to the check box may be output.

  The data file generation unit 24 generates a data file including text data of the plurality of text areas described above. This data file is, for example, a data file of a predetermined format such as a CSV (Comma-Separated Values) data file.

  For example, the text data described above is arranged in the data file in the order corresponding to the positions of the plurality of text areas described above. In this case, this order is specified based on, for example, the positions of a plurality of text areas in the horizontal or vertical direction in the document image. Alternatively, in this case, the data file generation unit 24 automatically detects the text object in the document image, identifies the attribute of the text object based on the user operation, and associates the attribute with the position of the text object. Template data is generated, and an attribute corresponding to the position of the text area described above (for example, an attribute associated with the position of the text object closest to the position of the text area) is specified based on the template data, Arrange the text data in the data file in a corresponding order.

  Also, for example, the text data described above is arranged in the data file in the order corresponding to the estimated character type. In this case, for example, text data is arranged in an order corresponding to the attribute indicated by the estimated character type, using a character type indicating an external attribute (such as date) as described above.

  The verification unit 25 verifies the presence or absence of the text of the specific text area (specified by the above-mentioned attribute, position, etc.) of the above-mentioned one or more text areas or the integrity of the text of the specific text area. You may For example, when the above-mentioned specific text area is an area for sealing a seal stamp, the verification unit 25 determines the presence or absence of a seal based on whether the text is specified by the text specification unit 23 and the determination result May be notified to the user, or if there is no image object in the specific text area, it may be determined that there is no seal. Also, for example, in the case where the above-mentioned specific text area is an area of a date, the verification unit 25 determines consistency based on whether the text specified by the text specifying unit 23 conforms to the date description format. The determination may be made and the determination result may be notified to the user.

  The machine learning processing unit 26 executes annotation processing for generating training data, and machine learning processing for machine learning the deep neural network of the character type estimation unit 22 with the training data generated by the annotation processing.

  For example, in the annotation processing, (a) a text object in a sample document image is automatically detected, and (b) a rectangular area surrounding only the text object for each text object is regarded as a text area (c The user operation assigns a character type in the text area, and (d) the pair of the document image (input data) and the position / size information of the text area in the document image and the character type (output data) is 1 Training data.

  In addition, in the annotation processing, the character type of a certain text region estimated by the character type estimation unit 22 is incorrect, and the correct character type and the position / size information of the text region (and its document image) when corrected by the user Position and size information of other text areas and character types) is taken as output data, and the document image at that time is taken as input data to include the process of generating training data, and the training data generated as such is included. Based on the above, the machine learning processing unit 26 may perform additional machine learning for the deep neural network of the character type estimating unit 22.

  The data file generation unit 24, the verification unit 25, and the machine learning processing unit 26 described above are not essential, and may be provided as needed.

  Next, the operation of the character type estimation system will be described. FIG. 4 is a flow chart for explaining the operation of the character type estimation system shown in FIG.

  First, the document image acquisition unit 21 acquires a document image (step S1).

  Next, the character type estimation unit 22 estimates the position and size information of each text area in the acquired document image, and the character type of the text in each text area (step S2). At this time, the character type estimation unit 22 inputs the acquired document image (raster image) as input data to the deep neural network using the machine-learned deep neural network, and outputs each as the output data of the deep neural network. Get position / size information and text type of text area.

  Then, the text specifying unit 23 executes character recognition processing for the character type estimated for the estimated text area, and specifies the text in the estimated text area as text data (step S3). At this time, by applying the character recognition process for the estimated character type, the character recognition rate is improved and the recognition error rate is decreased.

  After the character recognition process is performed on each text area, the verification unit 25 verifies the specific text area as described above, as necessary (step S4).

  Then, the data file generation unit 24 generates a data file including the text data specified for each text area, for example, in the CSV format (step S5). For example, using the data file generated in this manner, character information in the document image is registered in the database as text data.

  As described above, according to the above embodiment, the document image acquisition unit 21 acquires a document image, and the character type estimation unit 22 determines position / size information of one or more text areas in the document image, and The character type of the text in the text area is estimated for each of the one or more text areas, and the text identification unit 23 executes character recognition processing for the character type estimated for the estimated text area, and is estimated. Identify the text in the text area as text data. The character type estimation unit 22 is provided with a machine-trained deep neural network, the input data of the deep neural network is the above-mentioned document image, and the output data of the deep neural network is one or more texts of the above The above-mentioned position / size information and character type for each of the areas.

  As a result, appropriate character recognition processing is automatically performed on each text in the document without the user specifying the character type in advance, and extraction of character information in various documents is accurately performed.

  Note that various changes and modifications to the above-described embodiment will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing the intended advantages. That is, such changes and modifications are intended to be included in the scope of the claims.

  For example, in the above embodiment, the above system may use the estimated character type for other applications. For example, character recognition may not be performed for a specific character type.

  Further, in the above embodiment, the image data of the document image may be deleted from the system immediately after the above processing is completed.

  The present invention is applicable to, for example, a form reading system.

11 character type estimation program 21 document image acquisition unit 22 character type estimation unit 23 text specification unit 24 data file generation unit 25 verification unit 26 machine learning processing unit

Claims (6)

A document image acquisition unit that acquires document images;
Position / size information of one or more text areas in the document image, and a character type estimation unit for estimating a character type of text in the text area for each of the one or more text areas;
A text identification unit that executes character recognition processing for the character type estimated for the estimated text region to specify text in the estimated text region as text data;
The character type estimation unit comprises a machine-trained deep neural network,
The input data of the deep neural network is the document image,
The output data of the deep neural network is the position / size information and the character type for each of the one or more text regions,
Character type estimation system characterized by. And a data file generation unit for generating a data file including the text data of the plurality of text areas,
The text data is arranged in the data file in an order corresponding to the positions of the plurality of text areas.
The character type estimation system according to claim 1, characterized in that And a data file generation unit for generating a data file including the text data of the plurality of text areas,
The text data is arranged in the data file in an order corresponding to the character type.
The character type estimation system according to claim 1, characterized in that   The character type estimation system according to claim 1, further comprising: a verification unit that verifies presence / absence of text in a specific text area among the one or more text areas or integrity of text in the specific text area. . Document image acquisition step of acquiring a document image;
A machine type learned deep neural network, character type estimation for estimating position and size information of one or more text areas in the document image, and estimating a character type of text in the text area for each of the one or more text areas Step and
And C. performing a character recognition process for the character type estimated for the estimated text region to specify text in the estimated text region as text data.
The input data of the deep neural network is the document image,
The output data of the deep neural network is the position / size information and the character type for each of the one or more text regions,
Character type estimation method characterized by Computer,
Document image acquisition unit that acquires document images,
Position / size information of one or more text areas in the document image, and a character type estimation unit for estimating a character type of text in the text area for each of the one or more text areas, and the estimated text area Performing character recognition processing for the character type estimated for the character type to function as a text specifying unit for specifying text in the text region estimated as text data;
The character type estimation unit comprises a machine-trained deep neural network,
The input data of the deep neural network is the document image,
The output data of the deep neural network is the position / size information and the character type for each of the one or more text regions,
Character type estimation program characterized by.

Images