JP2023003887A - Document image processing system, document image processing method, and document image processing program - Google Patents

Abstract

To accurately identify an attribute of a cell in a table without using template data.SOLUTION: A table detection unit 22 detects a table in a document image and a cell in the table, and generates cell geometry data for the cell. A text object detection unit 23 detects a text object in the document image and generates text object geometry data for the text object. A cell attribute identification unit 25 identifies the text object in the cell based on the cell geometry data and the text object geometry data, generates, for each cell, node data including the cell geometry data, the text object geometry data of the text object in the cell, and text data of the text object in the cell, and executes predetermined classification processing on a node data set including the node data corresponding to the table to identify an attribute of the cell.SELECTED DRAWING: Figure 1

Description

The present invention relates to a document image processing system, a document image processing method, and a document image processing program.

In a certain form identification system, a form format table is created in advance by a user, and the form format table contains field information indicating the position, size, character type, etc. of a character recognition target area specified by the user. Character information (text data) in the form is obtained from the image data of the form image based on the form format (that is, field information) (see, for example, Japanese Unexamined Patent Application Publication No. 2002-100002).

A certain image recognition device cuts out a partial image from a target image, recognizes characters and numbers in the partial image, and executes extraction processing to extract characters and numbers that satisfy a predetermined condition from the characters and numbers (for example, See Patent Document 2). In the extraction process, the image recognition device, for example, determines whether or not the recognized characters include a predetermined bank name, and if the characters include the predetermined bank name, the characters and the Digits within a predetermined distance from a letter are extracted as pairs of bank name and account number.

JP 2016-48444 A JP 2020-170264 A

However, the form identification system described above uses template data that specifies the layout of documents such as forms (information on the position where each attribute is described, etc.), so in order to process multiple documents with different layouts, , template data must be created in advance for each layout, and complicated work is required in advance. Also, for a document whose layout is unknown, it is difficult to accurately detect the attribute value in the document image for a certain attribute with the above-described technique.

Further, in the image recognition apparatus described above, template data is unnecessary, but the character string to be extracted (the above-mentioned bank name) must be set in advance, and character strings that are not set are not extracted. In the image recognition apparatus described above, a number within a predetermined distance from the bank name is extracted as an account number. Even if the distance between the two is long, the desired character string may not be extracted correctly because the two may be related.

FIG. 5 is a diagram showing an example of a document image including a table. For example, as shown in FIG. 5, the document image 101 of the health checkup report includes a table 111 showing the results of the health checkup. The table 111 includes combinations of labels indicating inspection items and values that are inspection results of the inspection items. In the table 111, cells are arranged two-dimensionally regardless of the presence or absence of ruled lines, and labels and values corresponding to the labels (numerical values and character strings other than numerical values) are described in the cells. Label and value) may be unrelated even if the distance between them is short, or they may be related even if the distance between them is long.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and provides a document image processing system, a document image processing method, and a document image processing program for accurately specifying attributes of cells in a table without using template data. The purpose is to obtain

A document image processing system according to the present invention includes a table detection unit for detecting a table in a document image, detecting at least a cell in the table, and generating cell geometry data indicating the position and size of the cell; a text object detection unit for detecting a text object in a text object and generating text object geometric data indicating the position and size of the text object; a character recognition processor that generates data; (a) identifying a text object in a cell based on the cell geometry data and the text object geometry data; (c) for the node data set and a cell attribute identification unit that executes a predetermined classification process using the cell and identifies the attribute of the cell for each cell.

A document image processing method according to the present invention includes the steps of detecting a table within a document image, detecting at least cells within the table, generating cell geometry data indicating the position and size of the cell; detecting the object and generating text object geometry data indicating the position and size of the text object; performing character recognition processing on the text object to generate text data corresponding to the text object; a) identifying the text objects in the cell based on the cell geometry data and the text object geometry data; and (b) for each cell, the cell geometry data, the text object geometry data for the text objects in the cell, and generating node data containing the text data of the text objects in the cells; generating a node data set containing the node data corresponding to the table; and identifying attributes of the cell for each.

A document image processing program according to the present invention causes a computer to function as the above table detection unit, the above text object detection unit, the above character recognition processing unit, and the above cell attribute identification unit.

According to the present invention, a document image processing system, a document image processing method, and a document that accurately specify attributes of cells in a table without using template data that specifies the description position of attributes in the document image. An image processing program is obtained.

The above and other objects, features and advantages of the present invention will become further apparent from the following detailed description together with the accompanying drawings.

FIG. 1 is a block diagram showing the configuration of a document image processing system according to an embodiment of the invention. FIG. 2 is a diagram explaining node data generated in the document image processing system according to the embodiment of the present invention. FIG. 3 is a diagram for explaining classification processing in the document image processing system according to the embodiment of the present invention. FIG. 4 is a flow chart for explaining the operation of the document image processing system shown in FIG. FIG. 5 is a diagram showing an example of a document image including a table.

Embodiments of the present invention will be described below based on the drawings.

FIG. 1 is a block diagram showing the configuration of a document image processing system according to an embodiment of the invention. The document image processing system shown in FIG. 1 is composed of one information processing device (personal computer, server, etc.), but a processing unit, which will be described later, is distributed among a plurality of information processing devices capable of data communication with each other. good too. Also, such a plurality of information processing apparatuses may include GPUs (Graphics Processing Units) that perform specific computations in parallel.

The document image processing system shown in FIG.

The storage device 1 is a non-volatile storage device such as flash memory and hard disk, and stores various data and programs.

Here, the storage device 1 stores an image processing program 11, and also stores system setting data (such as coefficient setting values of a neural network used in each processing unit described later) as necessary. . The image processing program 11 may be stored in a portable computer-readable recording medium such as a CD (Compact Disk). In that case, for example, the image processing program 11 is installed from the recording medium to the storage device 1 . Also, the image processing program 11 may be a single program or an aggregate of a plurality of programs.

The communication device 2 is a device capable of data communication such as a network interface, a peripheral device interface, a modem, etc., and performs data communication with other devices as necessary. The image reading device 3 optically reads a document image from the document and generates image data (raster image data, etc.) of the document image. Note that the communication device 2 and the image reading device 3 are provided as necessary.

The arithmetic processing unit 4 is a computer equipped with a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. By executing it, it operates as various processing units.

Here, by executing the image processing program 11, the arithmetic processing unit 4 performs a document image acquisition unit 21, a table detection unit 22, a text object detection unit 23, a character recognition processing unit 24, a cell attribute identification unit 25, a data output It operates as a unit 26 and a machine learning processing unit 27 .

The document image acquisition unit 21 acquires a document image as image data such as raster image data. A document image can have one or more attributes (entry items etc.) in a table containing attribute labels (text such as headings) and attribute values (text such as numerical values and other character strings). For example, the document image acquisition unit 21 reads a document image as image data stored in the storage device 1, or acquires a document image as image data received by the communication device 2 via a communication channel such as a network. or acquires a document image as image data generated by the image reading device 3 .

A table detection unit 22 detects a table in an acquired document image without using template data, generates table geometry data indicating the position and size of the table, and detects at least cells in the table. and generate cell geometry data indicating the location and size of the cell.

In this embodiment, the table detection unit 22 detects at least one of rows and columns along with cells in the table, and provides row geometry data indicating the position and size of the row and column geometry data indicating the position and size of the column. At least one (here, both) of the scientific data is generated.

The table detection unit 22 detects tables in the document image, rows, columns, and cells in the tables, and generates geometric data for them according to an existing technique using a neural network.

The text object detection unit 23 detects text objects in the acquired document image without using template data, and generates text object geometric data indicating the position and size of the text objects.

Specifically, the text object detection unit 23 (a) detects text objects by excluding objects other than text (photo objects, graphic objects, ruled line objects, etc.) in the document image, and (b) detects each text object. Based on the position of , extract text objects by grouping them by "words".

Note that the text object detection unit 23 uses existing technology (for example, region separation processing, machine-learned deep neural network, etc.) to extract text objects in the document image.

The character recognition processing unit 24 performs character recognition processing on the detected text object (raster image) to generate text data (character code string) corresponding to the text object. Existing technology is used for this character recognition processing.

The cell attribute identification unit 25 (a) identifies the text object in the cell based on the cell geometry data and the text object geometry data described above, and (b) for each cell, determines the cell geometry data, the generating node data containing the text object geometry data for the text objects and the text data for the text objects in the cells, generating a node data set containing the node data corresponding to the table, and (c) for the node data set , a predetermined classification process is executed, and the attribute of the cell is specified for each cell.

Note that the positions of cells, rows, columns, and text objects in node data are
They are coordinate values in two-dimensional coordinates, and their size is the length at each coordinate in the two-dimensional coordinates. In addition, the position is indicated by the position of a predetermined portion (one of the four corners, the center, etc.) of the rectangular area of the cell, row, column, and text object, and the may be represented by the relative position of This relative position is derived from the absolute position of the table within the document image (the position in the table geometry data) and the absolute position of the cell, row, column or text object (the position in the original geometry data). .

Specifically, for each detected cell, the cell attribute specifying unit 25 places a text object (bounding box) specified from the text object geometric data in the cell region specified from the cell geometric data ) is included, the text object is determined to be the text object in the cell.

FIG. 2 is a diagram explaining node data generated in the document image processing system according to the embodiment of the present invention. FIG. 3 is a diagram for explaining classification processing in the document image processing system according to the embodiment of the present invention.

In this embodiment, as shown in FIG. 2, the node data for a cell consists of the cell geometry data for the cell, the text object geometry data for the text objects in the cell, and the text data for the text objects in the cell. In addition to the data, it further includes at least one (here, both) of the row geometry data of the row to which the cell belongs and the column geometry data of the column to which the cell belongs.

In this embodiment, as shown in FIG. 3, the cell attribute identification unit 25 inputs the node data set described above as input data to a machine-learned graph neural network (GNN), and the output data of the GNN is as an attribute of the cell in Existing GNN and its machine learning can be used.

Also, the attributes of a cell include at least the cell type of the cell, where the cell type is either label or attribute value (i.e., in this case, the cell is classified as either a label cell or an attribute value cell). is done). Note that the GNN output data for each node data is the probability (numerical value within the range of 0 to 1) for each possible value (label and attribute value here) of the cell type of the cell corresponding to the node data ), and based on the probability value, the cell type is determined to be one of the possible values (here, the label and the attribute value) of the cell type, for example, by classification using a threshold.

Note that the cell attribute identification unit 25 may perform clustering on the node data based on the feature amount indicated by the node data, classify the node data into clusters, and identify the attribute of the cell based on the clusters. . For example, the cell attribute identification unit 25 may identify the cell attribute by the above-described clustering instead of the GNN, or if the reliability of identifying the above-described cell attribute by the GNN is low, the GNN Alternatively, the above-described clustering may be used to identify the attributes of the cells.

For example, as this feature amount, a feature vector corresponding to node data (whole or specific part) generated according to an existing method such as Word2vec (Skip-Gram model) is used. We also collect a large number of combinations of node data and cell type values for that node data, and calculate the central value (mean of the feature vector) for each cell type value (here, label or attribute value) (each cluster center), and from the position indicated by the feature vector of the node data to be classified, the cell type value (here, label or attribute value) with the closest center value corresponds to that node data Selected as the cell type value.

The data output unit 26 adds a cell attribute corresponding to the node data to each node data, stores the node data set in a predetermined data format in the storage device 1, or transmits the node data set via the communication device 2. .

With this output data (node data set), for example, label cells and attribute value cells in a column can be specified, and label cells and attribute value cells in a row can be specified.

The machine learning processing unit 27 executes machine learning processing for performing machine learning of the GNN in the cell attribute specifying unit 25 described above. Note that the machine learning processing unit 27 described above is not essential, and may be provided as needed. Further, when the machine learning of the cell attribute identification unit 25 (GNN) has been completed, the machine learning processing unit 27 may not be provided.

Next, the operation of the document image processing system according to this embodiment will be described. FIG. 4 is a flow chart for explaining the operation of the document image processing system shown in FIG.

First, the document image acquiring section 21 acquires a document image (step S1).

Next, the table detection unit 22 detects tables in the document image without using template data, detects cells, columns, and rows in the tables, and extracts cell geometry data, column geometry data, and raw geometric data are generated (step S2).

The text object detection unit 23 also detects text objects in the document image without using template data and generates text object geometric data (step S2). The character recognition processing unit 24 performs character recognition processing on the detected text object to generate text data of the text object.

Note that the above-described processing by the table detection unit 22 and the above-described processing by the text object detection unit 23 may be executed in parallel. good.

Then, for each detected cell, the cell attribute specifying unit 25 specifies the text object in the cell based on the cell geometric data and the text object geometric data described above, and Generate node data including text object geometry data of the object and text data of the text object in the cell (step S3).

Next, for each table, the cell attribute identification unit 25 generates a node data set with node data corresponding to all cells detected in the table, and executes a predetermined classification process on the node data set. Then, the attribute of each cell is classified and the attribute of each cell (here, the cell type of label or attribute value) is specified (step S4).

Then, the data output unit 26 adds, for example, the attribute of each cell to the node data corresponding to the cell, and stores the node data set as output data in the storage device 1 in a predetermined data format for each table. , and transmitted by the communication device 2 (step S5).

In this way, attribute data for each cell is generated for each table in the document image.

In addition, the data output unit 26 receives a search request in a predetermined format, searches for an attribute value corresponding to the label specified by the search request in the node data set generated as described above, according to the search request, and You may make it output the combination of a label and an attribute value. For example, first, in the generated node data set, the cell, row, and column containing the label to be searched are identified, and the row or column containing the attribute value cell is identified in the row or column, If there is one attribute value cell in that row or column, that attribute value is specified as the attribute value corresponding to that label, and if there are multiple attribute value cells, those attribute values are assigned to that label. Identified as the corresponding attribute value, the row or column of the attribute value cell (if the attribute value cell is specified within the row containing the label to be searched, the column containing the label to be searched) If cells of attribute values are specified in the row), labels may be attached in association with those attribute values, respectively.

For example, in the node data set of the table 111 in FIG. 5, when "height" is specified as a search target, "161.0", "161.2", and "161.0", "161.2", and Three of "161.1" are detected, "161.0" has the label "this time", "161.2" has the label "previous", and "161.1" has the label "previous". Labeled.

Also, two labels may be designated as search target labels in a search request. In that case, an attribute value is detected in one of the two labels in the same way as described above, and among the detected attribute values, the label of the row or column to which the cell with that attribute value belongs matches the label of the other. are determined to be the attribute values corresponding to the two labels and are detected.

As described above, according to the above embodiment, the table detection unit 22 detects a table in a document image, detects at least cells in the table, and generates cell geometry data indicating the position and size of the cell. Generate. The text object detection unit 23 detects text objects in the document image and generates text object geometric data indicating the position and size of the text objects. The character recognition processing unit 24 performs character recognition processing on the text object to generate text data corresponding to the text object. The cell attribute identification unit 25 (a) identifies a text object in a cell based on the cell geometry data and the text object geometry data, and (b) identifies the cell geometry data and the text object in the cell for each cell. and node data containing the text data of the text objects in the cells; generating a node data set containing the node data corresponding to the table; A classification process is performed to identify, for each cell, the attributes of the cell.

This accurately identifies the attributes of the cells in the table without using template data. Also, since the attribute value corresponding to the label is detected without considering the distance (Euclidean distance) between the label and the attribute value, regardless of the distance (Euclidean distance) between the label and the attribute value, The exact combination of label and attribute value is specified.

Various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of its subject matter and without diminishing its intended advantages. It is therefore intended that such changes and modifications be covered by the claims.

For example, in the above embodiment, the image data of the document image may be deleted from the system immediately after the above processing is completed.

In the above embodiment, the number of nodes of the GNN input data is set to a predetermined value equal to or greater than the maximum value of the number of node data in the node data set (that is, the number of cells in the table). If the number of node data is less than the number of nodes, a fixed value is used as the missing node data, and the corresponding output data is discarded.

In the above embodiment, the attribute of a cell is the cell type, and the cell type is either a label or an attribute value. , etc. may be taken.

Further, in the above embodiment, rows and columns may not be detected and node data may not include row geometric data and column geometric data. Even so, the attribute value corresponding to the label is found by searching and finding the attribute value cell along the horizontal and vertical directions from the label cell position based on the cell position of the cell geometry data. can be detected.

INDUSTRIAL APPLICABILITY The present invention is applicable, for example, to recognition processing of document images such as forms.

4 Arithmetic processing unit (an example of a computer)
11 Image processing program (an example of a document image processing program)
22 Table detection unit 23 Text object detection unit 24 Character recognition processing unit 25 Cell attribute identification unit

Claims (7)

a table detector for detecting a table within a document image, detecting at least a cell within said table, and generating cell geometry data indicative of the position and size of said cell;
a text object detector for detecting text objects in the document image and generating text object geometry data indicating the location and size of the text objects;
a character recognition processing unit that performs character recognition processing on the text object to generate text data corresponding to the text object;
(a) identifying the text objects within the cells based on the cell geometry data and the text object geometry data; and (b) for each of the cells, the cell geometry data, the text within the cells. (c) generating node data including said text object geometry data for objects and said text data for said text objects in said cells, and generating a node data set including said node data corresponding to said table; a cell attribute identification unit that executes a predetermined classification process on the node data set and identifies an attribute of the cell for each of the cells;
A document image processing system comprising: 2. The cell attribute identification unit according to claim 1, wherein the node data set is input to a machine-learned graph neural network as input data, and the output data of the graph neural network is identified as the attribute of the cell. document image processing system. The cell attribute identification unit performs clustering on the node data based on the feature amount indicated by the node data, classifies the node data into clusters, and identifies the attribute of the cell based on the clusters. 2. The document image processing system according to claim 1. The table detection unit detects at least one of rows and columns together with cells in the table, and at least one of row geometric data indicating the position and size of the row and column geometric data indicating the position and size of the column. to generate
the node data of a cell further includes at least one of the row geometry data of the row to which the cell belongs and the column geometry data of the column to which the cell belongs;
4. The document image processing system according to any one of claims 1 to 3, characterized by: the attributes of the cell include at least a cell type of the cell;
the cell types include labels and attribute values;
5. The document image processing system according to any one of claims 1 to 4, characterized by: detecting a table within a document image, detecting at least a cell within said table, and generating cell geometry data indicative of the position and size of said cell;
detecting text objects in said document image and generating text object geometry data indicating the location and size of said text objects;
performing a character recognition process on the text object to generate text data corresponding to the text object;
(a) identifying the text objects within the cells based on the cell geometry data and the text object geometry data; and (b) for each of the cells, the cell geometry data, the text within the cells. (c) generating node data including said text object geometry data for objects and said text data for said text objects in said cells, and generating a node data set including said node data corresponding to said table; performing a predetermined classification process on the node data set to identify, for each cell, attributes of the cell;
A document image processing method comprising: the computer,
a table detector for detecting a table within a document image, detecting at least a cell within said table, and generating cell geometry data indicating the position and size of said cell;
a text object detector for detecting text objects in the document image and generating text object geometry data indicating the location and size of the text objects;
a character recognition processing unit that performs character recognition processing on the text object to generate text data corresponding to the text object;
(a) identifying the text objects within the cells based on the cell geometry data and the text object geometry data; and (b) for each of the cells, the cell geometry data, the text within the cells. (c) generating node data including said text object geometry data for objects and said text data for said text objects in said cells, and generating a node data set including said node data corresponding to said table; a cell attribute identification unit that executes a predetermined classification process on the node data set and identifies an attribute of each cell;
A document image processing program that functions as a

Images