JP7317886B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

Conventionally, there is a technique for automatically recognizing character strings present in a form using image data representing the form and converting the character strings into data including the correspondence between the character strings (for example, Japanese Unexamined Patent Application Publication No. 2002-200011). After recognizing the form, the operator usually performs post-processing such as checking the recognition result and correcting the recognition result as necessary.

JP 2009-122722 A

In the prior art including Japanese Patent Application Laid-Open No. 2002-200311, all character strings in a form are recognized including their corresponding relationships, and the recognition result is passed to subsequent processing. However, it is not always necessary to confirm all recognition results in subsequent processing.

Depending on the application, configuration, etc., there may be character strings on the form that do not need to be checked or that need not be checked. In subsequent processing, a situation is conceivable in which the recognition result of such a form is confirmed. In addition, it is conceivable that a situation may arise in which only character strings with a relatively high degree of importance need to be confirmed in subsequent processing due to the operator's burden or time constraints. In any situation, the operator has to search out the recognition result to be confirmed from among the recognition results and confirm it, which makes it difficult to perform subsequent processing quickly.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and provides a technique that enables an operator to more quickly perform appropriate subsequent processing for confirming the recognition result of a form depending on the situation. With the goal.

In order to achieve the above object, an information processing device according to one aspect of the present invention includes:
Using data of an image representing a form, recognizing a plurality of character strings in which one or more characters exist in the form, and character specifying position information of each of the recognized character strings in the image. column recognition means;
predetermined two character strings out of the plurality of character strings existing in the form using the recognition result of each of the plurality of character strings by the character string recognition means and the position information of the plurality of character strings; Correspondence determining means for determining the correspondence between
an identifying means for identifying a set of two character strings satisfying a predetermined condition, from among the two predetermined character strings for which correspondence has been determined by the correspondence determining means.

According to the present invention, it is possible for the operator to more quickly perform appropriate subsequent processing for confirming the recognition result of the form depending on the situation.

1 is a diagram for explaining an outline of a first service that can be realized by a form recognition device according to an embodiment of an information processing device of the present invention; FIG. FIG. 10 is a diagram illustrating an example of extraction results of a table, a cell, and a character string area extracted from a form image; FIG. 10 is a diagram showing a display example of various extraction results of a table, cells, and character string areas in a display area; FIG. 10 is a diagram illustrating an outline of a second service that can be realized by the form recognition device according to one embodiment of the information processing device of the present invention; FIG. 10 is a diagram showing an example of a set of two character strings specified from a form; 1 is a diagram showing a configuration example of an information processing system constructed using a form recognition device according to an embodiment of an information processing device of the present invention; FIG. 1 is a block diagram showing an example of a hardware configuration of a form recognition device according to an embodiment of an information processing device of the present invention; FIG. 1 is a functional block diagram showing an example of a functional configuration implemented on a form recognition device according to an embodiment of an information processing device of the present invention; FIG. 5 is a flow chart showing an example of cell detection processing executed by the form recognition device according to the embodiment of the information processing device of the present invention; FIG. 10 is a diagram showing examples of components identified as possibly not properly connected to other components; FIG. 4 is a diagram illustrating a first example of the content of processing executed in step S13; FIG. FIG. 10 is a diagram illustrating a second example of the content of processing executed in step S13; FIG. FIG. 3 is a diagram showing an example of a form image, cells detected from the form image, and grouping of the detected cells; 6 is a flow chart showing an example of perimeter acquisition processing executed by the form recognition device according to the embodiment of the information processing device of the present invention; FIG. 10 is a diagram showing an example of cell detection results and node information obtained by converting the detection results; FIG. 10 is a diagram showing an example of grouped node information and an example of extraction of the perimeter of a table including cells from the node information; FIG. 3 is a functional block diagram showing an example of a functional configuration actually implemented as a part of a key-value extractor on the form recognition device according to one embodiment of the information processing device of the present invention; FIG. 10 is a diagram illustrating an example of various information generated by a graph information generation unit; FIG. FIG. 10 is a diagram showing a detailed example of a graph constructing unit; FIG. 10 is a diagram illustrating another specific example of a set of character strings having a key-value relationship;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram for explaining an overview of a first service (hereinafter referred to as "this first service") that can be realized by a form recognition device according to an embodiment of an information processing device of the present invention.

The first service is a service that can be realized by a form recognition device (see FIG. 6, which will be described later). The service provider develops, for example, dedicated application software (hereinafter referred to as "dedicated application"), and provides the first service to individuals or organizations that have purchased the dedicated application. Here, it is assumed that the organization purchased the dedicated application. The organization is hereinafter referred to as the "purchasing company", and the person who actually uses the dedicated application within the purchasing company is referred to as the "operator".

For example, an operator installs a dedicated application on an arbitrary information processing device in the purchasing company, and can use the information processing device as a form recognition device, that is, use the first service. can.

A form recognition device is used to digitize a form, that is, to digitize a form. For example, when a paper-medium form is read by a scanner, the scanner outputs image data representing the form. An image representing a form is hereinafter referred to as a "form image" as appropriate. Data of an image representing a form is called "form image data". The form recognition device receives the form image data output from the scanner.

The form recognition device provided by the first service has a function of individually extracting each of the cells, tables, and character string regions present in the form by analyzing such form image data. ing. Such a function is hereinafter referred to as a "layout analysis function". That is, it can be said that the first service is to provide a form recognition device capable of exhibiting a layout analysis function.

Here, a cell is a portion surrounded by ruled lines on a form. Most of the character strings on a form usually exist in cells.

A character string area is an area in which a string of one or more characters representing some meaning exists.
A character string is spatially or content-distinguishable from other character strings. A character is a symbol formed using at least one of lines and dots. Hiragana, katakana, kanji, numbers, alphabets, and various marks are all characters.

A table is a range in which one or more cells forming a group are identified by the cells and their positions, and the identified one or more cells exist. Thereby, a plurality of cells forming a table structure are regarded as one table as forming a group.
In the first service, when one or more character strings are arranged in a range where no cell exists, one character string area is extracted as a table. A table may be extracted as a group consisting of one or more cells and one or more character string areas (see FIG. 2, which will be described later).

Specifically, for example, according to the first service, the form recognition device performs the following processing using the form image data by exerting the layout analysis function.
That is, the form recognition device recognizes a cell from the ruled lines present in the form image, and extracts the cell by specifying the position of the recognized cell in the form image.
Further, the form recognition device identifies one or more cells that form a group in the form image based on the one or more cells and their positions thus extracted, and lists the range in which the identified one or more cells exist. Extract as By specifying one or more cells forming a group in the form image, the position of the table in the form image is also specified.
Further, the form recognition device recognizes a character string area existing in the form, and extracts the character string area by specifying the position of the recognized character string area in the form image.

Further, a specific example of the layout analysis function will be described below with reference to FIGS. 1 to 3. FIG.

FIG. 1 shows a display screen DS on which the processing result of the layout analysis function by the first service can be displayed.
As shown in FIG. 1, the display screen DS has a display area DS1, a display area DS2, and a display area DS3.

The display area DS1 is a display area for allowing the operator to select form image data (file) to be subjected to layout analysis.
That is, one or more form image data are input in advance to the form recognition device in the form of a file. The operator selects form image data to be subjected to layout analysis from among the one or more form image data in the display area DS1.

The display area DS2 is an area for displaying the form image represented by the form image data selected as the analysis target in the display area DS1. Further, the display area DS2 is an area in which the layout analysis results, that is, the extraction results of the cell, table, and character string areas are superimposed and displayed on the form image independently of each other.
Thumbnail images of various images including the form image (original image) to be analyzed are displayed in the upper portion of the display area DS2. In addition to the original image, the images displayed as thumbnail images include a binarized image of the original image, a saliency map image focusing on the ruled lines on the original image, and a saliency map image focusing on the ruled lines and character string areas on the original image. Contains a saliency map image. The two saliency map images are images generated using, for example, a DNN (Deep Neural Network).

The display area DS3 is an area for displaying the details of the layout analysis result displayed in the display area DS2.

In the upper portion of the display area DS3, an area in which a button BT1 labeled "table" inside, a button BT2 labeled "cell" inside, and a button BT3 labeled "text" inside are arranged. DS31 (hereinafter referred to as "button display area DS31") is arranged.
The button BT1 is a button for performing an instruction operation to superimpose the extraction result of the table on the form image to be processed.
The button BT2 is a button for performing an instruction operation to superimpose a cell extraction result on the form image to be processed.
The button BT3 is a button for performing an instruction operation to superimpose the extraction result of the character string area on the form image to be processed.

The operator can confirm only the extraction result desired to be confirmed in the display area DS2 by clicking, for example, the button of the extraction result desired to be confirmed among these buttons BT1 to BT3. Here, the extraction result for which confirmation is desired means the extraction result of at least one or more of the table, cell, and character string area.
Note that all the three buttons BT1 to BT3 are in the selected state immediately after the layout analysis is executed. In this case, the operator can confirm all the extraction results of the table, cell, and character string area (text).

Immediately below the button display area DS31, a tab T1 labeled "table", a tab T2 labeled "cell", a tab T3 labeled "text", and a tab T3 labeled "recognition result (1)” is arranged therein. Below these four tabs T1 to T4, a detailed display area DS32 reserved for displaying the details of the analysis results is arranged.

The tab T1 is a tab for instructing operation to display the details of the extraction result of the table as a result of the layout analysis. In the first service, as details of the table extraction results, the location information and an excerpt of its image are displayed for each table.

The positional information is expressed assuming xy coordinates with the upper left point of the form image as the base point. The shape of one table is basically assumed to be a rectangular shape surrounded by two lines parallel to the x-axis and two lines parallel to the y-axis. As a result, in this first service, as position information, each distance on the x-axis and y-axis between the base point and the upper left point of the table, the width that is the length on the x-axis, and the distance on the y-axis The height, which is the length of , is extracted. All these units are pixels. Information included in this position information is the same for cells and character string areas.
Hereinafter, the direction parallel to the x-axis is also referred to as the "horizontal direction", and the direction parallel to the y-axis is also referred to as the "vertical direction". Further, unless otherwise specified, the positional information is used to mean the position and shape of the area.

The "cell" tab T2 is a tab for instructing display of the details of the cell extraction result as the analysis result. In the first service, location information of each cell and an extract of an image of each cell are displayed.
The "text" tab T3 is a tab for instructing display of the details of the extraction result of the character string area as the analysis result. In the first service, the position information of each character string area and an excerpt of the image of each character string area are displayed.
The "recognition result (1)" tab T4 is a tab for instructing display of the recognition result of the character string (text) as the analysis result. In the first service, the recognition result of each character string and the excerpt of the image of each character string are displayed in association with each other.

By clicking one of these tabs T1 to T4, the operator can confirm only the details of the desired analysis result in the detail display area DS32. For example, as shown in FIG. 1, by clicking on the "recognition result (1)" tab T4, the operator confirms the recognition result of each character string present on the form by comparing it with the image of the character string. be able to. In addition to the positional information, the extracted image is also displayed, so the operator can more quickly and easily check the analysis results including the character string recognition results.

A "save data" button BT5 is arranged below the detail display area DS32 in the display area DS3. This button BT5 is a button for the operator to instruct the saving of the analysis result including the character string recognition result and various extraction results. As shown in FIG. 1, when this button BT5 is clicked, a pop-up menu DS321 is displayed for allowing the operator to select a data storage format. A "data save" button BT6 is arranged in the menu DS321. Accordingly, the operator can save the analysis result including the recognition result of the character string in the desired storage format by clicking the "save data" button BT6 after selecting the desired storage format.

FIG. 2 is a diagram illustrating an example of extraction results of tables, cells, and character string regions extracted from a form image.
A total of five tables TB are extracted from the extracted portion of the form image FI shown in FIG. One cell is extracted as one table TB in all four of the five tables TB. The remaining one is a table in which a total of 11 cells CE and one character string region TX vertically adjacent to one of them are grouped together.

As shown in FIG. 2, all four tables TB having only one cell CE are located away from any of the other cells CE and the character string area TX. Therefore, even one such cell CE is extracted as one table TB on the assumption that there is no logical correspondence with any of the other cells CE and the character string region TX.

On the other hand, in the table TB including 11 cells CE and one character string region TX, each cell CE is adjacent to at least one other cell CE horizontally or vertically. Therefore, the 11 cells CE are treated as elements forming a table structure and grouped.

One character string region TX grouped in the same group as the 11 cells CE is not arranged within the cell CE. However, this character string area TX is adjacent to two cells CE located above and to the right respectively. Due to the positional relationship, the character string area TX is considered to have a corresponding relationship with any character string arranged in the two cells CE. As a result, the character string regions TX that are not arranged in the cell CE are grouped with 11 cells CE and made an element of one table TB. In fact, there is a semantic correspondence between the character string "deadline" in the character string area TX and the character string "December 20, 2014" in the cell CE located to the right. do. In addition, hereinafter, "TX" is also used to indicate the character string itself.

In FIG. 2, a table TB having 11 cells CE and one character string area TX as elements is taken as an example. showing.
The cell CE indicating the location information is the cell CE located second from the top among the cells CE located on the left side of the table TB. Therefore, the distance on the x-axis included in the position information matches the distance on the table TB, and the distance on the y-axis does not match the distance on the table TB.

The character string area TX indicating the position information is the character string area TX existing in the cell CE located second from the top among the cells CE located on the left side of the table TB. Therefore, in the position information, each distance on the x-axis and the y-axis are all greater than the distance of the cell CE. However, the width and height are all smaller than the width and height of the cell CE in which the character string region TX is arranged.

FIG. 3 is a diagram showing a display example of various extraction results of tables, cells, and character string areas in the display area.
As described above, there are three types of extraction results: table TB, cell CE, and character string area TX. Three buttons BT1-BT3 arranged in the button display area DS31 make it possible to selectively display one of them. In addition, in the first service, three types of extraction results are displayed simultaneously as an initial display. For this reason, in FIG. 3, (A) to (D) show different extraction results to be displayed. Specifically, FIG. 3A shows an example of an initial display, that is, a case where each extraction result of the table TB, cell CE, and character string region TX is displayed. Similarly, FIGS. 3B to 3D show examples of displaying the extraction results of the table TB, cell CE, and character string region TX, respectively. As a result, in FIG. 3A, all buttons BT1 to BT3 of "table", "cell" and "text" are in the selected state. In FIGS. 3B to 3D, only one of the three buttons BT1 to BT3 is selected.

As shown in FIGS. 3A to 3D, each extraction result of the table TB, cell CE, and character string area TX is indicated by a rectangular frame. The color of the frame is made different depending on the table TB, cell CE, and character string TX. Thereby, the operator can visually recognize each extraction result of the table TB, the cell CE, and the character string area TX by the color of the frame.

As described above, in the first service, one cell CE that is located away from other cells CE is extracted as one table TB (see FIG. 2). In the first service, similarly, one character string area TX that is positionally separated from other cells CE and other character string areas TX is also extracted as one table TB. As a result, as shown in FIG. 3(A), each of the character strings in the character string area TX, ``FY2017 Income and Expenditure Statement'' and ``Income Section'', is formed into one table TB and one cell. Also extracted as CE. The character string in the character string area TX, "from April 1, 2017 to. March 31, 2018" is "from. April 1, 2017" to ". Month 31st” is divided into two clauses, and the regions in which each clause exists are extracted as one character string region TX, one cell CE, and one table TB.

In the first service, as shown in FIGS. 3A to 3D, the operator individually confirms desired extraction results by clicking any of the three buttons BT1 to BT3. be able to. Therefore, the operator can easily confirm the extraction results of the table TB, cell CE, and character string area TX.

Not all of the string TX can always be properly recognized. When a handwritten character string TX exists on a form, the probability of properly recognizing the handwritten character string TX is normally relatively low. Not all character strings TX can be properly recognized even in a form printed by converting data into an image using a PC (Personal Computer) or the like. This is because there may be a problem that occurred during printing, dirt or dust adhering to the form when the form was converted into image data, writing on the form, damage that occurred on the form, and the like.

For this reason, it is common for the operator to at least confirm the recognition result of the character string and perform subsequent processing to correct any errors in the recognition result after converting the form into data, that is, digitizing it. Subsequent processing often targets a large number of forms, and the formats of the target forms are often varied. As the types of form formats increase, the time required for the operator to confirm the recognition results of character strings and their corresponding relationships increases. For this reason, it is desirable for the purchasing company to enable operators to perform subsequent processing more efficiently.

According to the layout analysis as described above, the first service can save analysis results including the results of extracting the cell CE, the character string area TX, and the table TB, in addition to the character string recognition results. Thereby, the operator can visually recognize each extraction result of the extracted cell CE, the character string area TX, and the table TB.

Most of the character strings on the form exist in cells. Therefore, even if either the cell CE or the character string area TX is visible, the operator can more easily and quickly check the recognition result of the character string TX and the correspondence between the character strings TX. .

On the other hand, when the table is made visible, it is possible for the operator to check the recognition result of the character string TX and the corresponding relationship for each table. As described above, the table TB is an area larger than one cell CE (or one character string area TX). Divide logically. As a result, it becomes easier and faster for the operator to check the recognition result for each table in the form average. As a result of dividing the table TB, the number of character strings TX to be considered is reduced, and the table structure becomes more recognizable, making it easier to predict appropriate correspondences between character strings TX. This is because If an appropriate correspondence relationship between character strings TX can be predicted, the operator can easily determine not only whether the recognition result of the character string TX is appropriate, but also whether the correspondence specified between the character strings TX is appropriate. and can be verified more quickly.
As such, the operator can more reliably and efficiently perform subsequent processing.

FIG. 4 is a diagram for explaining an outline of a second service (hereinafter referred to as "this second service") that can be realized by the form recognition device according to one embodiment of the information processing device of the present invention.

This second service is also a service that can be realized by a form recognition device (see FIG. 6, which will be described later). The service provider, for example, develops a dedicated application similar to the first service, and provides the second service to individuals or organizations who have purchased the dedicated application. Here too, it is assumed that it is the organization that purchased the dedicated app, etc., and that organization is hereinafter referred to as the "purchasing company", and the person who actually uses the dedicated app within the purchasing company is referred to as the "operator". .

For example, by installing a dedicated application in any information processing device in the purchasing company, the operator can use the information processing device as a form recognition device, that is, use the second service.

The form recognition device provided by the second service recognizes character strings TX present in a form by performing processing using form image data, and identifies a set of two character strings TX that satisfy a predetermined condition. It has the function to Such a function is hereinafter referred to as a "key-value extraction function". That is, it can be said that the second service is to provide a form recognition device capable of exhibiting a key-value extraction function.

Here, the key is a character string TX that is logically located on the upper side in terms of the correspondence between two character strings TX. A value is a character string TX that is logically located on the lower side in terms of the correspondence relationship. A key is usually a character string TX that represents an identifier corresponding to the specific content represented by the character string TX that is a value.

Specifically, for example, according to the second service, the form recognition device performs the following processing using the form image data by exerting the key-value extraction function.
That is, the form recognition apparatus uses data of an image representing a form to recognize a plurality of character strings TX in which one or more characters are consecutively present in the form image, and recognizes each form of the plurality of recognized character strings TX. Identify location information in an image.
Furthermore, the form recognition device recognizes two predetermined character strings TX out of the plurality of character strings TX present in the form, using the recognition results of each of the plurality of character strings TX and the position information of the plurality of character strings TX. determine the correspondence between
Also, the form recognition device identifies a set of two character strings TX that satisfy a predetermined condition, among the two predetermined character strings TX for which the correspondence relationship has been determined.

Further, a specific example of the layout analysis function will be described below with reference to FIGS. 4 and 5. FIG.

FIG. 4 shows, as an overview of the second service, an example of the flow of recognizing a character string TX from a form image FI and identifying a set of two character strings TX.
In the second service, the character string TX recognized from the form image FI is treated as a node ND of the graph. This graph connects the nodes ND with a line, and each line indicates the correspondence between two nodes ND, that is, the edge ED. Accordingly, specifying a set of two character strings TX corresponds to extracting two nodes ND connected by an edge ED from the graph.

In the graph shown on the right side of FIG. 4, the node ND is assigned one of ND1 to ND3 as a symbol, and the edge ED is assigned one of ED1 to 3 as a symbol. ND1 to ND3 represent attributes of different nodes ND. ED1-3 represent different combinations of attributes of the nodes ND connected by the edges ED. If there is no need to distinguish between them, "ND" is used as the node code, and "ED" is used as the edge code.

Specifically, ND1 represents a node classified as a key, ND2 represents a node classified as a value, and ND3 represents a node classified as others. Also, ED1 represents an edge connecting two nodes ND1 and ND2 classified as key and value, respectively. Similarly, ED2 represents an edge that connects two nodes ND1 and ND3 classified as keys and others, respectively, and ED3 represents an edge that connects two nodes ND that are combined other than those.

FIG. 5 is a diagram showing an example of a set of two character strings specified from a form image.
FIGS. 5A and 5B show an example of a set of two character strings TX specified from the form image for each different form.
In the form image shown in FIG. 5(A) , as a set of two character strings TX, the character string TX11 is “assessment year”, the character string TX13 is “Heisei 26”, and the character string TX12 is “target year”. and "Heisei 26" which is the character string TX14 is specified. In addition, a set of two character strings TX of "notification number" and "6200100001", "period/month" and "July", and "deadline" and "December 20, 2014" is specified. .

Similarly, in the form image shown in FIG. 5B, the character string TX21 "total amount" and the character string TX23 "53,999" are identified as a set of two character strings TX. In addition, the character string TX22 "receipt amount" and the character string TX24 "53,999" are also identified as a set of two character strings TX. In addition, "customer number" and "072-0000028-005", "period of use" and "August 1, 2014 to September 30, 2014", "waste water amount" and "44", "usage Amount" and "53,999", "reminder fee" and "0", "already delivered" and "0", "delivery deadline" and "March 31, 2015" are also two character strings TX specified as a set.

Generally, a form recognition device recognizes all character strings TX present on a form and saves the recognition result. Subsequent processing by the operator basically confirms all of the recognition results.

However, the importance of character strings TX present on a form may differ. For example, character strings forming a tabular structure often have a corresponding relationship with other character strings TX, and the need for confirmation by the operator is usually relatively high. On the other hand, since the character string TX such as the name of the form usually has no correspondence with other character strings TX, the necessity of confirmation is usually relatively low.

As for the correspondence between character strings TX, there may be correspondence between three or more character strings TX. Of these, the character string TX positioned at the end, more specifically, the character string TX positioned at the right end or the bottom end is often the most important in terms of the table structure. A character string TX that has a direct correspondence with the character string TX located at the end often has a relatively high degree of importance. A character string TX with a higher degree of importance needs to be confirmed appropriately.

Also, due to the operator's burden or time constraints, a situation may occur in which only the character string TX with a relatively high degree of importance needs to be confirmed in subsequent processing.
If only the recognition result of the character string TX with a relatively high degree of importance is to be confirmed in subsequent processing, the operator must find the recognition result to be confirmed from among all the recognition results and confirm it. It becomes difficult to process quickly.

On the other hand, in the second service, it is possible to limit the character strings TX to be subjected to subsequent processing from among the character strings TX recognized on the form image. It is possible to store only the set of two strings TX that are considered particularly important, or to separate them separately.
As such, it becomes easier for the operator to properly confirm the pair of the two character strings TX. It also allows the operator to better avoid making serious mistakes in subsequent processing. As a result, the operator can more quickly perform appropriate follow-up processing according to the situation or the like.

FIG. 6 is a diagram showing a configuration example of an information processing system constructed using a form recognition device according to an embodiment of the information processing device of the present invention.
The form recognition device 1 is an information processing device prepared by the purchasing organization, and the purchased dedicated application is installed. The form recognition apparatus 1 is connected to a scanner 2 capable of reading an image of a form, digitizing it, and outputting the form image data obtained by the digitization. Thus, the information processing system has a configuration in which the form recognition device 1 and the scanner 2 are connected. Here, it is assumed that the form recognition device 1 is used by the operator for subsequent processing.

FIG. 7 is a block diagram showing an example of the hardware configuration of the form recognition device according to one embodiment of the information processing device of the present invention.
The form recognition device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input/output interface 15, an output section 16, and an input section. 17 , a storage unit 18 , a communication unit 19 and a drive 20 .

The CPU 11 executes various processes according to programs recorded in the ROM 12 or various programs loaded from the storage unit 18 to the RAM 13 . Various programs include the above two dedicated applications. The CPU 11 executes the various programs so that the information processing device functions as the form recognition device 1 .
The RAM 13 also stores data necessary for the CPU 11 to execute various processes. The data also includes various programs executed by the CPU 11 .

The CPU 11 , ROM 12 and RAM 13 are interconnected via a bus 14 . An input/output interface 15 is also connected to this bus 14 . An output unit 16 , an input unit 17 , a storage unit 18 , a communication unit 19 and a drive 20 are connected to the input/output interface 15 .

The output unit 16 is configured to include a display such as a liquid crystal, for example. The output unit 16 displays various images under the control of the CPU 11 . Various images include a display screen DS as shown in FIG.
The input unit 17 includes, for example, various hardware buttons such as a keyboard. As a result, various information can be input via the input unit 17 . The input unit 17 may include a plurality of input devices such as pointing devices and touch panels.

The storage unit 18 is, for example, a hard disk device or an auxiliary storage device such as an SSD (Solid State Drive). Data having a large amount of data are stored in the storage unit 18 .
A communication unit 19 enables communication with the scanner 2 .

A drive 20 is provided as required. A removable medium 31 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory card can be attached to and detached from the drive 20 . When the removable medium 31 on which the program is recorded is attached to the drive 20 , the program can be stored in the storage section 18 . Also, the removable medium 31 can be used as a copy destination or a transfer destination of various data stored in the storage unit 18 .

As a result of controlling the hardware resources of the form recognition device 1 by various programs, the form recognition device 1 provides the first and second services to the operator belonging to the company that purchased the dedicated application. can do. Various processes to be described later are realized by the CPU 11 executing various programs.

FIG. 8 is a functional block diagram showing an example of a functional configuration realized on the form recognition device according to one embodiment of the information processing device of the present invention.

As shown in FIG. 8, in the CPU 11 of the form recognition device 1, when it is assumed that two dedicated applications are executed, the layout analysis section 101 and the key-value extraction section 102 function. Accordingly, the CPU 11 corresponds to a computer that controls the form recognition device 1, which is an information processing device.

The layout analysis unit 101 exhibits a layout analysis function that can be provided by the first service. The form image data transmitted from the scanner 2 is received by the communication section 19 and input from the communication section 19 to the CPU 11 . When form image data is input to the CPU 11, the active layout analysis unit 101 includes a cell extraction unit 111, a table extraction unit 112, a character string recognition unit 113, and a display unit 113, as shown in FIG. A control unit 114 and an input control unit 115 function.

The cell extraction unit 111 uses the form image data to recognize cells existing in the form, and extracts the cells by specifying the positions of the recognized cells in the form image.
Specifically, for example, the cell extraction unit 111 extracts the cells CE by detecting ruled lines existing on the form image FI represented by the form image data. For each extracted cell CE, the cell extraction unit 111 also extracts the range in which the cell CE exists in the form image as a cell area, and extracts the position information of the cell area (see FIG. 2). As described above, each distance on the xy axis of the position information is expressed by extracting a rectangular shape that is regarded as a form on the form image, and using the upper left point of the extracted rectangular shape as a base point. be. This also applies to the character string area TX and the table TB.

The character string recognition unit 113 recognizes the character string area TX existing on the form image FI and identifies the position (position information) of the recognized character string area TX in the form image FI, thereby extracting the character string area TX. do. The character string TX is a sequence of one or more characters. The character string recognition unit 113 recognizes a plurality of character strings TX present in the form using the image data representing the form image FI. When functioning as part of the layout analysis unit 101, only recognition of the character string area TX and specification of its position may be performed. That is, recognition of the character string TX is not essential.

The table extraction unit 112 identifies one or more cells CE forming a group in the form image FI based on the one or more cells CE and the positions extracted by the cell extraction unit 111, and the identified one or more cells CE are Extract the existing range as a table TB. A table TB may be extracted from the character string area TX or including the character string area TX. Therefore, the table extraction unit 112 extracts the table TB by referring to the location information extraction result for each cell CE by the cell extraction unit 111 and the location information extraction result for each character string region TX by the character string recognition unit 113. do. The table extraction unit 112 also extracts the position information of the table TB for each extracted table TB (see FIG. 2).

The cell extraction unit 111, the table extraction unit 112, and the character string recognition unit 113 generate information necessary for displaying the display screen DS as shown in FIG. Information obtained by the cell extraction unit 111, the table extraction unit 112, and the character string recognition unit 113 is stored as an analysis result in the analysis result storage unit 182 secured in the storage unit 18. FIG. Form image data transmitted from the scanner 2 and input to the CPU 11 via the communication section 19 is stored in the image storage section 183 secured in the storage section 18 .

The display control unit 114 performs control to display an image in a form in which the positions of the extracted cells CE, table TB, and character string TX can be visually recognized. Accordingly, the display control unit 114 can cause the output unit 16 to display a display screen DS as shown in FIG. When displaying the display screen DS on the output unit 16, the display control unit 114 reads the corresponding form image data from the image storage unit 183 and the corresponding analysis result from the analysis result storage unit 182, and displays the display screen DS. Generate data for By outputting the generated display data to the output unit 16, the output unit 16 displays the display screen DS.

The input control unit 115 recognizes and processes the operation performed by the operator on the input unit 17, and performs control according to the recognition result. In other words, the input control unit 115 corresponds to the operator's click operation on the three buttons BT1 to BT3, the "data save" buttons BT5 and BT6, and the four tabs T1 to T4, and the operator's selection of the save format on the menu DS321. do. Accordingly, the operator can confirm the desired extraction result on the display area DS2 by clicking any one of the three buttons BT1 to BT3. Further, the operator can confirm the details of the desired extraction result or the recognition result of the character string TX on the display area DS3 by clicking any of the four tabs T1 to T4. Therefore, the input control unit 115 issues an instruction to the display control unit 114 according to the button or tab whose operation has been recognized.

When the operator selects the save format in the menu DS321 and then clicks the "save data" button BT6, the input control unit 115 instructs the display control unit 114 according to the click operation. Thereby, the display control unit 114 causes the output unit 16 to display the display screen DS in the state before the menu DS321 is displayed, for example.

On the other hand, the input control unit 115 passes information representing the selected storage format to the table extraction unit 112 and instructs storage of the analysis results in that storage format.
According to the instruction, the table extraction unit 112 stores the analysis result in the analysis result storage unit 181 secured in the storage unit 18 in the storage format selected by the operator.

The key-value extraction unit 102 exhibits a key-value extraction function that can be provided by the second service. In the enabled key-value extraction unit 102, as shown in FIG. 8, a character string recognition unit 113, a correspondence determination unit 116, and an identification unit 117 function.

When the key-value extraction unit 102 is enabled, the character string recognition unit 113 uses the data of the image representing the form image FI to extract a plurality of character strings TX in which one or more characters are consecutively present in the form. Along with recognition, position information in each image of the recognized character strings TX is specified.

The correspondence determination unit 116 uses the recognition result of each of the plurality of character strings TX by the character string recognition unit 113 and the position information of the plurality of character strings TX to identify the character strings TX among the plurality of character strings TX present in the form image FI. , determine the correspondence between two given strings TX (see FIG. 5).

The identifying unit 117 identifies a set of two character strings TX that satisfy a predetermined condition among the two predetermined character strings TX whose correspondence relationship is determined by the correspondence determining unit 116 (see FIG. 5). A set of two strings TX in a key-value relationship is an example.

9 to 16, an outline example of the processing executed to provide the first service will be described below.

FIG. 9 is a flow chart showing an example of cell detection processing executed by the form recognition device according to one embodiment of the information processing device of the present invention.
The cell extraction unit 111 shown in FIG. 8 is implemented by the CPU 11 executing this cell detection process. For this reason, the cell extracting unit 111 is assumed to be the entity that executes the processing.

First, in step S11, the cell extraction unit 111 converts the form image FI into a one-channel grayscale image and binarizes it, and divides the ruled lines existing on the binarized form image FI into horizontal line components and vertical line components. To separate.

In this embodiment, the separation of ruled lines into horizontal line components and vertical line components is performed by using morphological transformation on the binarized form image FI. Structural elements of prepared morphological filters are used to separate the horizontal line component and the vertical line component of the ruled line. Morphological transformation is also used to thicken line components, supplement line components, etc. through enlargement/reduction processing of horizontal line components and vertical line components.
Morphological transformation is performed using DNN. The two saliency map images displayed as thumbnail images in the upper portion of display area DS2 (FIG. 1) include ruled lines. The ruled line is identified using a horizontal line component and a vertical line component separated using DNN. Therefore, DNN is used to generate two saliency map images.

In step S11, the cell extraction unit 111 separates only the horizontal line component and the vertical line component by taking the logical product of each pixel between the horizontal line component and the vertical line component and the original form image FI. It also generates finished images.

The scanner 2 does not necessarily read the form properly. Moreover, there is a possibility that distortion or the like has occurred in the read form. For this reason, in practice, it is necessary to perform preprocessing such as rotation of the form image FI and correction of distortion occurring in the form image FI before executing the process of step S11.

Next, in step S12, the cell extraction unit 111 processes the two separated images to detect horizontal line components and vertical line components surrounding the rectangular area. The cell extraction unit 111 also extracts intersection information, which is position information about points where the detected components intersect.
In addition, the cell extraction unit 111 groups the rectangular regions surrounded by the detected components within a range considered to be connected, that is, adjacent. As a result, rectangular areas that are adjacent to each other at a distance equal to or less than a predetermined threshold value in, for example, up, down, left, or right directions are combined into one group. The extracted intersection information is referenced for grouping rectangular regions.
In addition, the cell extracting unit 111 also performs a process of identifying a component among the detected components that may not be properly connected to other components, and manipulating the identified component as necessary. .

FIG. 10 is a diagram showing examples of components identified as possibly not properly connected to other components.
In a ruled line, as shown in FIG. 10, a portion La of a component L of a horizontal line or a vertical line may protrude from another component L with which it intersects. In the first service, the protruding portion La is handled in consideration of the possibility that the protruding portion La should be connected to another component L existing in the protruding direction. Specifically, when there is another component L having a part La protruding toward the part La in the direction in which the part La protrudes, the two parts La He is trying to replace with one component L which partially contains La. Due to the replacement, the two parts La are treated as a ruled line connecting them. Hereinafter, the part La will be referred to as the "protruding portion La".

In such a replacement, it is a condition that the two different components L have facing protruding portions La. This is because it is highly probable that the middle portion of the component L disappeared for some reason, and instead of the component L, two opposing protruding portions La were generated. In the first service, the presence of the two facing protruding portions La is regarded as evidence indicating the high possibility. Due to these conditions, all the protruding portions La shown in FIG. 10 are not connected to other components L.

For a cell CE with rounded corners, the intersection information of the rounded corners will indicate a position away from that component L, even for neighboring cells CE that share the same component L. Therefore, it is difficult to appropriately perform grouping using intersection information.

However, if the ruled line is separated into horizontal and vertical line components, the rounded corner portion of the cell CE is ignored, and the two components L that lead to the rounded corner may not both be properly connected to the other component L. is identified as component L where the nature is considered. These two components L intersect with other components L on the extension of the component L. For this reason, both of these two components L are regarded as components L to be connected with each other L, and are changed to components L extending to the point of intersection with the other L components. As a result, all rounded corners of cell CE are shaped into square corners. With such shaping, grouping using intersection information can be performed appropriately regardless of the corner shape of the cell CE.

One component L may originally be detected as a plurality of components L because some of the components L have disappeared in the middle or dust adhered to the form at the time of reading. In such multiple components L, there are other components L facing in the same direction. Therefore, these multiple components L are replaced with one component L.

When a new point at which the component L intersects is generated by such an operation on the component L, the intersection information of that point is extracted. Therefore, by executing the process of step S12, most of the intersection information of the rectangular areas that are considered possible cells CE are extracted.

In step S13, the cell extraction unit 111 converts the information of each horizontal line component and vertical line component into a horizontal line component without height and a vertical line component without width, respectively. The cell extraction unit 111 aggregates the information after the conversion as grid coordinate information, determines whether or not there is a ruled line in each element of the grid divided at the intersection of the horizontal line component and the vertical line component, and stores the determination result in the grid. Save as ruled line presence/absence data for each element of .

FIG. 11 is a diagram for explaining a first example of the content of the processing executed in step S13.
Both the horizontal line component LH and the vertical line component LV are originally thick components. The horizontal line component LH is converted to a component without height, and the vertical line component LV is converted to a component without width. As a result, as shown in FIG. 11, grid data representing a grid formed by thin horizontal line components and vertical line components is created.

By assuming horizontal line components and vertical line components with no thickness, as shown in FIG. 11, it is possible to avoid different coordinate information due to differences in thickness. Thus, for vertical line components, for example, two vertical line components with different positions on the y-axis and different thicknesses can have the same x-coordinate, which simplifies the processing.

Each element of the grid is a component L divided by a horizontal line component and a vertical line component. For example, in a vertical line component whose x-coordinate value is _x1 , three parts divided by four horizontal line components are each an element. That is, y-coordinate values between y ₀ and y ₁ , between y ₁ and y ₂ , and between y ₂ and y ₃ are elements.

The presence/absence of ruled lines is determined for each element. In the example shown in FIG. 11, only elements with an x coordinate value of x ₁ and y coordinate values of y ₁ to y ₂ are determined to have no ruled lines, and "0" representing the determination result is ruled line presence/absence data. ing. All the other elements are determined to have ruled lines, and "1" representing the determination result is ruled line presence/absence data.

In the element whose x-coordinate value is x ₂ and whose y-coordinate value is between y ₂ and y ₃ , the vertical line component LV does not exist in the entire range. However, the vertical line component LV is present in most of the elements. Therefore, it is determined that this element also has a ruled line.

Thus, in the first service, the presence or absence of ruled lines is determined for each grid element. Therefore, even if an element has one or more components and one component does not cover the entire element, it is determined that the element has ruled lines, and the one or more components are Complemented by ingredients.

FIG. 12 is a diagram illustrating a second example of the contents of the processing executed in step S13.
As described above, in step S12, the protruding portion La protruding from the other component L is identified. FIG. 12 shows an example in which two protruding portions La protruding in the same horizontal direction, one protruding portion La protruding in the opposite direction, and one protruding portion La protruding in the vertical direction are specified. showing.

As shown in FIG. 12, each overhang La extends in the overhang direction to a component L that initially exists in the overhang direction and is perpendicular to that direction. Thereby, the component L detected during the process of step S12 is manipulated as shown in FIG. 12 by the process of step S13. A portion where the two corner components L do not intersect is manipulated so that they are connected at right angles by determining whether or not there is a ruled line for each element.

The reason why each protruding portion La is extended in the protruding direction is that there is a component L that may be connected in the protruding direction. In other words, it is possible that the range of the table TB, that is, part of the component L, which is the outer circumference, could not be detected.

Through such an operation, it can be expected that the components that make up the ruled line will be appropriately interpolated. Also, the rectangular area that can be regarded as the cell CE and the range where the rectangular areas are gathered can be made closer to a rectangular shape.

A part having a table structure is often rectangular as a whole. Therefore, making the rectangular area and the range where the rectangular areas are gathered closer to a rectangular shape is expected to be effective in extracting the cell CE and the table TB more appropriately.

In step S14, the cell extraction unit 111 refers to the grid coordinate information and ruled line presence/absence data for each grid element, and detects all rectangular regions including regions that are not included in other rectangular regions as cells CE. Location information is extracted from all the detected cells CE. After detecting the cell CE and extracting the location information from the detected cell CE in this way, the cell detection process ends.

FIG. 13 is a diagram showing an example of a form image, cells detected from the form image, and grouping of the detected cells.
As shown in FIG. 13, all rectangular areas surrounded by ruled lines on the form image FI and not included in other rectangular areas are detected as cells CE. The detected cells CE are grouped according to the positional relationship including shape, and the cells CE forming one group are treated as constituent elements of the table TB. A closed ruled line CL formed by a ruled line surrounding a non-rectangular area is not detected as a cell CE and is excluded.
By the exclusion, the valid ruled line is determined. Thereby, ruled lines on the two saliency map images are also determined. The character string area TX arranged on one side is extracted and specified separately from the ruled lines.

FIG. 14 is a flow chart showing an example of perimeter acquisition processing executed by the form recognition device according to one embodiment of the information processing device of the present invention.
The table extraction unit 112 shown in FIG. 8 is implemented by the CPU 11 executing this outer circumference acquisition process. For this reason, the table extraction unit 112 is assumed to be the entity that executes the processing.
In the cell detection process, the cells CE are detected and the cells CE are grouped. Therefore, in the perimeter acquisition process, the range of the table TB is extracted by specifying the perimeter of the set of grouped cells CE.

First, in step S21, the table extraction unit 112 converts the position information of the cell CE into node information. Assuming that four corners are specified from the positional information of each cell CE, and horizontal and vertical lines are drawn that intersect at the specified corners, the node information indicates whether the ruled line representing the cell CE is a horizontal line or a vertical line. This is information that indicates the position of the intersection point.

FIG. 15 is a diagram showing an example of cell detection results and node information obtained by converting the detection results.
By detecting or grouping the cells CE as shown on the left side of FIG. 15, it is assumed that horizontal lines and vertical lines that intersect at the corners of each cell CE are drawn as grid lines GD. A node P is a point where a ruled line representing a cell CE intersects a horizontal line or a vertical line of the grid line GD. The node information is position information representing the position of the node P. FIG.

Each node P is connected to at least part of a ruled line representing the cell CE. There are at least two of them at each node P. Some of them are hereinafter collectively referred to as "link LK".

At each node P, link information representing each link LK connected to the node P is generated together with the node information. Link information is information indicating the positional relationship between a node P and another adjacent node P. In this first service, as information indicating the positional relationship, the presence or absence of an adjacent node P and, if the node P exists, the distance to that node P are stored for each of the left, right, up, and down directions of the node P. I have to. The distance is expressed in units of pixels. Based on the link information, it is possible to specify the presence or absence of another node P adjacent in the four directions of left, right, up and down, and if there is another adjacent node P, the distance to that node P can be specified.

In step S22, the table extraction unit 112 groups the generated node information and link information according to the results of grouping when detecting the cell CE.

After that, in step S23, the table extracting unit 112 extracts the perimeter of the range of the grouped cells CE as the perimeter of the table TB by referring to the node information and link information of the group for each group. After extracting the perimeters of all the tables TB, the perimeter acquisition process ends.

FIG. 16 is a diagram showing an example of grouped node information and an example of extracting the perimeter of a table including cells from the node information.
A node P is assumed to specify the perimeter of the table TB. The specification of the outer circumference of the table TB is performed by a search in which the node P of interest is sequentially moved from the node P serving as the starting point to another adjacent node P. This search ends when it returns to the already searched node P.

In the first service, among the nodes P specified from the grouped node information, the search is performed with the leftmost and uppermost node P as a starting point. Priority is given to the direction of travel to the next node P. As shown in FIG. Regarding the order of priority, when viewed from the traveling direction in which the current node P is moved, the left direction has the highest priority. The next highest priority is the current direction of travel, that is, forward, the next highest priority is rightward, and the lowest priority is backward. As a result, in the example shown in FIG. 16, the node P marked with "P" is the starting point, and a search is performed to move only the node P located on the outer circumference, and the outer circumference of the table TB is extracted.

FIG. 17 is a functional block diagram showing an example of a functional configuration actually implemented as a part of the key-value extractor on the form recognition device according to one embodiment of the information processing device of the present invention.

As shown in FIG. 17, in the CPU 11 of the form recognition device 1, when the key-value extraction unit 102 functions, the graph information generation unit 121 and the graph construction unit 122 function. The graph information generation unit 121 and graph construction unit 122 include the functions of the correspondence determination unit 116 and the identification unit 117 .

The graph assumed in the second service has a data structure consisting of nodes ND and edges ED. The node ND here is the recognized string TX. The edge ED is data representing the correspondence between the nodes ND, that is, the relationship. Hereinafter, "node" is used to mean the recognized character string TX.

The graph information generation unit 121 receives from the character string recognition unit 113 an image (data) DT1 representing the form image FI, position information DT2 of the character string area in which the recognized character string TX exists, and recognition result DT3 of the character string TX. do.

In the graph information generation unit 121, as shown in FIG. 17, a graph modeler unit 1211 and a feature calculation unit 1212 function.
The graph modeler unit 1211 refers to the image DT1 and the position information DT2 of the character string areas TX, and generates relative position information between the character string areas TX, that is, distance information.
From the recognition result DT3 of the character string TX, the feature calculation unit 1212 generates attribute information representing the attribute of each character string TX and NLP information that is the result of performing natural language processing (NLP) on the character string. do.

FIG. 18 is a diagram illustrating an example of various information generated by the graph information generation unit.
The relative positional information between the character string areas TX is information representing spatial features. The graph modeler unit 1211 generates the recognized character string TX, more precisely, relative position information between adjacent character string regions TX for each character string region in which the character string TX exists. In the second service, as shown in FIG. 18, it is divided into four directions, up, down, left, and right, and position information up to the adjacent character string area TX is generated for each direction. Specifically, for vertically adjacent character string areas TX, position information is generated by dividing the distance between them by the height of the target character string area TX. Positional information is generated by dividing the distance between adjacent character string areas TX by the width of the target character string area TX.

By dividing the distance between the character string areas TX by the size (shape) of the target character string area TX, the relative position information between the character string areas TX can be obtained by determining the distance between the character string areas TX as the character string It is normalized according to the size of the area TX. The distance between the character string areas TX is important information for specifying the correspondence between nodes. By normalizing the distance, it becomes possible to more appropriately specify the correspondence between the character string areas TX regardless of the size of the character string areas TX.

The feature calculation unit 1212 determines the attribute of the character string TX according to a predetermined classification, and generates the attribute as Boolean information. The character string TX is classified, as shown in FIG. 18, into, for example, date, numerical value, or others different from them. Boolean attribute information is information that indicates, for each classification, whether or not it belongs to that classification.

The NLP information is information obtained by performing natural language processing on the recognized character string TX. This NLP information makes it possible to treat a plurality of words separated by blanks as one character string TX, as shown in FIG. 18, for example. In addition, "From April 1, 2017 to March 31, 2018" shown in FIG. as two strings TX of .

Various information generated by the graph modeler unit 1211 is passed to the graph construction unit 122 .
In the graph construction unit 122, as shown in FIG. 17, a feature extraction unit 1221, a node classification unit 1222, a node extraction unit 1223, an edge classification unit 1224, and a connection unit 1225 function. The feature extraction unit 1221, the node classification unit 1222, and the edge classification unit 1224 use MLP (MultiLayer Perceptron).

The feature extraction unit 1221 has a function of repeatedly learning the feature amount of each node ND. For example, if the node ND is an image of the character string TX "total amount", the node ND is recognized as the character string TX meaning "total amount" by comparing with the teacher data.

The node classification unit 1222 classifies each node ND using the feature amount extracted from each node ND, and generates the classification result as a node classifier. Here, each node ND is categorized as one of key, value, and other. This node classifier corresponds to the attribute information of the node ND.

Some of the nodes ND (character strings TX) on the form image FI can be classified into both keys and values. In the second service, a node ND that can be classified into both a key and a value is classified into only one of them by learning using learning data. Thereby, only the set of two nodes ND to be identified, that is, the set of node ND1 and node ND2 can be identified.

A node ND for which a correspondence between nodes ND, that is, a relationship should be specified is a node ND classified as a key or a value. The node ND3 classified as others is not included in the targets for identifying the correspondence relationship. Accordingly, the node extraction unit 1223 extracts only the nodes ND1 and ND2 classified as keys or values from among the nodes ND.

The edge classification unit 1224 generates an edge ED by predicting a node ND having a correspondence relationship with only a node ND classified as a key or value. Since the nodes ND for determining the correspondence relationship, that is, for predicting the edge ED, are limited to those classified as keys or values, the amount of calculation and the required Both the amount of memory required is reduced.

The node classifier of each node ND and the prediction result of the edge ED are passed to the combining unit 1225 . As a result, the connecting unit 1225 extracts only the edge ED1 that connects the two nodes ND1 and ND2 whose node classifiers are KEY and VALUE from the predicted (determined) edges ED. By extracting this edge ED1, a set of two nodes ND1 and ND2 whose node classifiers are key and value is specified.

FIG. 19 is a diagram illustrating a detailed example of a graph constructing unit;
The feature extraction unit 1221 is a three-layer NN (Neural Network), as shown in FIG. The feature extraction unit 1221 receives from the graph information generation unit 121 a graph representing a set of nodes ND and a set of edges ED, the number of nodes, the number of edges, node attributes, node labels, an adjacency matrix, a mapping matrix, and the like. be

A node label is information that enables identification of a node ND, for example. A mapping matrix is a multi-dimensional matrix representing the relationship between nodes ND, and is updated according to the prediction result of edge ED. The node attribute is logical type information, as shown in FIG.

As described above, the feature extraction unit 1221 extracts the feature amount of each node ND and identifies the meaning. As a result of this identification, the node extracting unit 1223 is notified of the node ND classified as other by the node classifier.

The node classification unit 1222 is also a three-layer NN, as shown in FIG. The node classifier 1222 generates a node classifier for each node using node attributes and node labels. Here, each node ND is classified into one of key, value, and others, so the node classifier is represented by a numerical value of 0-2. 0 represents the key, 1 the value, and 2 the others. A node classifier for each node is passed to the node extraction unit 1223 .

The node classifier 1222 performs learning using learning data for generating node classifiers. This learning data employs graph theory, for example, the node ND is classified into either a key or a value, and the strength of the relationship between them is annotated as link information. Thereby, the node classifier 1222 considers other nodes ND and generates a node classifier for each node ND. A node ND that can be classified into both a key and a value is classified into only one of them. Further, when a character string TX is classified as a key, all character strings TX corresponding to synonyms are also classified as keys.

In the node extraction unit 1223, as shown in FIG. 19, a node deletion unit 12231 and a node selection unit 12232 function.
Nodes ND classified as others are notified to the node deletion unit 12231 , and the node classifier of each node is passed to the node deletion unit 12231 . Accordingly, the node deletion unit 12231 deletes the node ND classified as other from among all the nodes ND. As a result, the node selection unit 12232 is notified only of the nodes ND classified as keys or values.

The node selection unit 12232 selects the node ND notified from the node deletion unit 12231 and notifies the edge classification unit 1224 of the selection result.
As a result, the edge classification unit 1224 identifies nodes ND that are considered to have a correspondence relationship with only the nodes ND classified as keys or values, and updates the mapping matrix.

The mapping matrix is, for example, a matrix that indicates the presence or absence of correspondence between nodes ND with 0 or 1. 0 indicates that there is no correspondence, and 1 indicates that there is a correspondence. Accordingly, the edge classifying unit 1224 updates the value of the element corresponding between the nodes ND where the correspondence is recognized from 0 to 1.

The edge classification unit 1224 is a two-layer NN that implements a natural language processing function. As a result, the edge classification unit 1224 considers that an edge ED exists between two nodes ND having a key-value relationship in the table structure (see FIG. 5). However, the two nodes ND are not necessarily classified as keys and values, respectively. Therefore, the connecting unit 1225 refers to the mapping matrix and the node classifier, and identifies the edge ED1 connecting the two nodes ND1 and ND2, which are indicated by the node classifier as key and value. Thereby, the connecting unit 1225 extracts only the set of two nodes ND1 and ND2 connected by the edge ED1. All the sets of these two nodes ND1 and ND2 are stored in the identification result storage unit 184 of the storage unit 18 as identification results.

An embodiment of the present invention has been described above, but an embodiment to which the present invention is applied may be, for example, as follows.

For example, some forms have a plurality of nodes ND2 classified as values for a node ND1 (character string TX) classified as keys. For example, in the form shown in FIG. 20, the character string TX31 of "full name" corresponds to the node ND1 classified as a key, and the character strings TX41 and TX42 of "Taro Yamada" correspond to the values having a corresponding relationship with the node ND1. Corresponds to the classified node ND2. Therefore, in addition to the character string TX41, the character string TX42 may also be associated with the character string TX31. Similarly, the character strings TX43 and TX44 may be associated with the character string TX32, the character strings TX45 and TX46 with the character string TX33, and the character strings TX47 and TX48 with the character string TX34.

The character strings TX43 to TX47 are also associated with the character string TX41 "Taro Yamada". For this reason, the set of character string TX41 and character strings TX43 to TX47 may be associated together. This set can be treated as structured data.
Further, although the form image data representing the form image FI is obtained from the scanner 2, the form image data may be obtained from a terminal connectable via a network. In other words, the device that acquires the form image data is not particularly limited. Accordingly, the device for displaying the display screen DS as shown in FIG. 1 is not particularly limited.

The screen configuration of the display screen DS is not limited to that shown in FIG. 1 either. For example, a display area for always displaying the extraction result of the table TB may be secured, and each extraction result of the cell area and the character string area may be selectively displayed in another display area. The operator may correct the recognition result of the character string TX displayed in the detail display area DS32.
Although NN is used to extract a set of two character strings TX in a key-value relationship, NN may not be used. However, when NN is used, there is an advantage that it becomes possible to correspond to various forms relatively easily with prepared learning data.
Multiple types of ruled lines may exist within the range of the table TB. Multiple types of ruled lines are multiple types of ruled lines that are distinguished by any of the following: different thicknesses, different numbers of lines forming one ruled line, different colors, and the like. There is usually some intention in making the ruled lines different. For this reason, the determination result of the ruled line type may be reflected in the extraction of the table TB or the determination of the correspondence between the character strings TX (nodes ND).
The form has a tabular structure. Therefore, the target for extracting the table TB and determining the correspondence between the character strings TX is the form. As a result, any object having a table structure can be targeted. In other words, the object may be something other than a form.

Although it has been explained that the first service and the second service are provided by different dedicated applications, these services may be provided by one dedicated application. In FIG. 8, it is assumed that the dedicated application capable of providing the second service uses a part of the functions of the dedicated application capable of providing the first service (the function that realizes the character string recognition unit 113). are doing.

In summary, the information processing apparatus capable of providing the first service, to which the present invention is applied, only needs to have the following configuration, and various embodiments can be adopted.
That is, an information processing device capable of providing the first service (for example, the form recognition device 1 shown in FIG. 6)
A cell extracting means (for example, a cell shown in FIG. 8) extracts a cell by recognizing a cell existing in the form using image data representing the form and specifying the position of the recognized cell in the image. an extraction unit 111);
Based on the one or more cells and their positions extracted by the cell extracting means, one or more cells forming a group in the image are identified, and a range in which the one or more identified cells exist is defined as a table. a table extracting means for extracting (for example, a table extracting unit 112 shown in FIG. 8);
Character string extraction means (for example, the character string recognition unit shown in FIG. 8) extracts a character string region by recognizing a character string region existing in the form and specifying the position of the recognized character string region in the image. 113) and
Information processing device.

This allows the operator to confirm the extraction results of the cell, character string area, and table. Many cells have text placed inside them. As a result, it is possible to confirm the positions where most of the character strings exist from the cell extraction results.
The extraction result of the table makes it possible to perform subsequent processing for confirming the recognition result of the character string and the corresponding relationship for each table. Since the confirmation can be made on a table-by-table basis, the number of character strings to be aware of at one time can be reduced, making it easier to confirm the correspondence between character strings.
As such, the operator can more easily and quickly perform appropriate follow-up processing.

The information processing device includes
Execution of control to display the image in a form in which the positions of the cells, the table, and the character strings extracted by the cell extracting means, the table extracting means, and the character string extracting means can be visually recognized. Display control means (for example, the display control unit 114 shown in FIG. 8) can be further provided.
This allows the operator to confirm the positions of the cells, tables, and character string areas from the displayed image.

The information processing device includes
The display control means is
Based on an operation by an operator, control is executed to display the image in a form in which one or more positions of the table, the cell, and the character string area are selectively visible. be able to.
This allows the operator to selectively check any position of the cell, table, and character string area from the displayed image. Therefore, it becomes easier to confirm the position.

Further, the information processing apparatus capable of providing the second service, to which the present invention is applied, only needs to have the following configuration, and various embodiments can be adopted.
That is, an information processing device capable of providing the second service (for example, the form recognition device 1 shown in FIG. 6)
Using data of an image representing a form, recognizing a plurality of character strings in which one or more characters exist in the form, and character specifying position information of each of the recognized character strings in the image. a string recognition means (for example, a character string recognition unit 113 shown in FIG. 8);
predetermined two character strings out of the plurality of character strings present in the form using the recognition result of each of the plurality of character strings by the character string recognition means and the position information of the plurality of character strings; Correspondence determination means (for example, correspondence determination unit 116 shown in FIG. 8) for determining the correspondence between
specifying means (for example, specifying unit 117 shown in FIG. 8) for specifying a set of two character strings satisfying a predetermined condition among the two predetermined character strings for which the correspondence relationship is determined by the relationship determining means;
Information processing device.

As a result, the operator can easily perform confirmation limited to a set of two character strings of particularly high importance among the character strings present in the form. It is possible to avoid the need to search for the recognition result to be confirmed among the character string recognition results. From a different point of view, it becomes easier to avoid checking recognition results of character strings that are not required or are relatively infrequently checked.
As such, the operator can take appropriate follow-up actions more quickly.

The information processing device includes
The correspondence relationship determination means adopts only character strings having any one of one or more predetermined attributes among the plurality of character strings as each of the two predetermined character strings, and It is possible to determine the correspondence between two predetermined character strings.
This makes it possible to specify a desired pair of two character strings through the attributes of the character strings to be adopted.

The information processing device includes
The predetermined condition can include a condition where one character string is a key and the other character string is a value.
This allows the set of two strings that have that relationship to be identified.

1 form recognition device, 2 scanner, 11 CPU, 12 ROM, 13 RAM, 14 bus, 15 input/output interface, 16 output section, 17 input section, 18 storage section, 19 communication section, 20 drive, 31 removable, edia, 111 Cell extractor 112 Table extractor 113 Character string recognizer 114 Display controller 115 Input controller 116 Correspondence determining unit 117 Identifying unit

Claims (4)

Using data of an image representing a form, recognizing a plurality of character strings in which one or more characters exist in the form, and character specifying position information of each of the recognized character strings in the image. column recognition means;
determination of an attribute based on the result of recognition of each of the plurality of character strings by the character string recognition means, the position information of the plurality of character strings, and each of the plurality of character strings, and generation using the determined attribute Correspondence determination means for determining a correspondence relationship between two predetermined character strings among the plurality of character strings present in the form using a classifier;
identifying means for identifying a set of two character strings that satisfy a predetermined condition from among the two predetermined character strings for which the correspondence has been determined by the correspondence determining means;
the attributes represent any of dates, numbers, and others;
The classifier may be any one of a key logically located on the upper side of the correspondence between the two predetermined character strings, a value logically located on the lower side of the correspondence, and others. represents
Information processing equipment. The correspondence determining means adopts, from among the plurality of character strings, only character strings having one of the predetermined one or more classifiers as each of the two predetermined character strings. to determine the correspondence between the two given strings;
The information processing device according to claim 1 . The correspondence determining means uses the position information to generate relative position information between the character strings, and uses the generated relative position information to identify the correspondence between the two character strings. and determining two character strings for which one of the classifiers is the key or the value among the two character strings for which the correspondence relationship is specified as the two predetermined character strings,
The identifying means identifies a set of the two character strings in which the classifier of one character string is the key and the classifier of the other character string is the value.
The information processing apparatus according to claim 1 or 2. In the information processing method executed by the information processing device,
Using data of an image representing a form, recognizing a plurality of character strings in which one or more characters exist in the form, and character specifying position information of each of the recognized character strings in the image. a column recognition step;
determining an attribute based on the recognition result of each of the plurality of character strings by the processing of the character string recognition step, the position information of the plurality of character strings, and each of the plurality of character strings; and using the determined attribute a correspondence determining step of determining a correspondence between two predetermined character strings among the plurality of character strings present in the form using the generated classifier;
a specifying step of specifying a set of two character strings that satisfy a predetermined condition from among the two predetermined character strings for which the correspondence has been determined by the processing of the correspondence determining step;
the attributes represent any of dates, numbers, and others;
The classifier may be any one of a key logically located on the upper side of the correspondence between the two predetermined character strings, a value logically located on the lower side of the correspondence, and others. represents
Information processing methods.

Images