JP7252818B2 - Data extraction system from digital documents - Google Patents

Description

The present invention relates to a technique for extracting data such as graphs and tables from digital documents.

Data analysis using AI (Artificial Intelligence) is a service that analyzes data such as newspaper articles, papers, and corporate financial statements published as digital documents and provides valuable information to users. It's becoming As a preparation for data analysis, it is necessary to convert a large amount of digital documents collected from the Web etc. into a format that is easy to use. Digital documents are provided in various file formats, such as Hyper Text Markup Language (HTML) and Extensible Markup Language (XML). Digital documents in these formats contain not only the data itself, but also the structure of the data. Contains meta information. Analyze the structure of such digital documents, extract only the data to be analyzed, such as the title of the digital document, text, author, publication date, table, graph, etc., and store it in a storage device such as a data base (DB). It is necessary to perform the conversion process for data analysis as data preparation for data analysis.

Currently, many digital documents are distributed as files in Portable Document Format (PDF), and analysis of PDF files is required in data analysis. When analyzing data, it is necessary to analyze the structure of the PDF file and extract the data to be analyzed as data preparation. Data such as tables and graphs are highly likely to contain important information, and extraction of table and graph data is strongly desired.

A PDF file has a complicated format such as a mixture of text and binary, and it is difficult to analyze the data structure as it is. In addition, a PDF file consists of objects (text, images, etc.) in the document page and coordinate information that indicates where the object exists in the page, and indicates what kind of data each object is. Information is not embedded. For example, information such as whether the text in the document page is the text of the document or table data is not embedded in the PDF file. Difficulty identifying and extracting data in documents.

As a conventional technology, there is a method of converting the text data of the PDF file into a format that facilitates analysis of the data structure, and then extracting the table data. Extraction of tabular data from PDF files is done through table detection, layout recognition, and in-table text extraction. Table detection is to detect a table from document pages of a PDF file, and layout recognition is to recognize which row and column each cell of the table belongs to. The table text extraction is a process of extracting the detected text in the table. The extracted text is saved in a storage device in a format such as an HTML table representation that reproduces the structure (layout) of the original table, or a DB table.

One of the conventional techniques is a method of converting a PDF file into an image file and analyzing the data structure. Analyze image information such as table ruled lines (table ruled lines, etc.) to detect tables and recognize table layouts. The text in the table is extracted by OCR (Optical Character Recognition/Reader). Patent Document 1 discloses a technique of extracting a table as image data and extracting the table data by image analysis.

Another method is to convert the PDF file into a text file with coordinate information and analyze the data structure. By using a text extraction tool such as PDFMiner (https://github.com/euske/pdfminer/), it is possible to obtain a text file with the coordinates within the page assigned to the text in the PDF document. Coordinates are given in a certain unit (per unit, per sentence, etc.). By analyzing the text with coordinate information and analyzing the positional relationship between the texts, table data detection and layout recognition are performed. Non-Patent Document 1 and the like can be cited as an example.

US20150093021A1

Martha et.al., “TAO: System for Table Detection and Extraction from PDF Documents”, FLAIRS Conference, 2016

The method of extracting tabular data by image analysis disclosed in Patent Document 1 has a problem of erroneously detecting an object that is not a table on a page as a table. Further, in recognizing the layout of a table, there is a problem that the layout of the table cannot be accurately recognized for a table lacking image information, such as a table containing only horizontal ruled lines. In addition, there is a problem that misrecognition of OCR text occurs when extracting text in a table.

On the other hand, in the method of analyzing the text with coordinate information of Non-Patent Document 1 and extracting the table data, it is not possible to use image information such as the ruled lines of the table. However, there is a difficult problem.

Accordingly, an object of the present invention is to improve the quality of data analysis for digital documents.

A preferred aspect of the present invention includes a conversion processing unit that converts a digital document containing an object in a document page and coordinate information of the object into image data and text data with coordinate information, and an object to be extracted from the text data. A data extraction system from a digital document comprising a text data analysis unit for extracting associated captions and an image analysis unit for detecting objects to be extracted from image data based on the extracted caption information.

Another preferred aspect of the present invention is a method for extracting data from a digital document for extracting a desired object from the digital document, comprising text data generation processing for generating text data with coordinate information from the digital document, keywords From a digital document comprising: a caption detection process for detecting a caption of a desired object from text data by searching; and an object extraction process for extracting a desired object from image data of a page of the digital document where the caption is detected data extraction method.

Improve the quality of data analysis for digital documents. Further specific problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is a block diagram showing the hardware configuration of a data extraction device according to an embodiment; FIG. 3 is a logical block diagram showing the logical configuration of the data extraction program according to the embodiment; FIG. FIG. 4 is a flow diagram showing the process when detecting captions for tables; FIG. Table diagram showing an example of a text file with coordinate information when converting a PDF file into a text file with coordinate information. FIG. 10 is a table diagram showing an example of a text pattern used to distinguish between a table caption and a text when detecting a table caption. FIG. 11 is a table diagram showing an example of a table caption detection result sent to the image analysis unit; FIG. 4 is a flow diagram showing processing when detecting a table and performing layout recognition; FIG. 4 is an explanatory diagram showing how to limit the range of image analysis when detecting a table; FIG. 10 is a flow diagram showing processing when layout recognition is executed for a table in which cells are represented in a grid pattern by ruled lines; FIG. 10 is a flowchart showing processing when executing layout recognition of a table lacking ruled line information; FIG. 10 is a flowchart showing processing when recognizing rows and columns of a table that includes merged cells and lacks ruled line information; FIG. 4 is a table diagram showing an example of a table for histogram analysis; FIG. 10 is a chart showing an example of results of cell recognition performed by histogram analysis; FIG. 10 is an explanatory diagram showing an example of correct layout recognition in table layout recognition; FIG. 11 is an explanatory diagram showing an example of a case where a layout is recognized erroneously in layout recognition of a table; FIG. 4 is an explanatory diagram showing a method of analyzing the text of a PDF document that exists in a coordinate information text file and recognizing the layout of a table. FIG. 4 is a table diagram showing the structure of coordinate information of a table extracted by a data extraction program; FIG. 4 is a flow diagram showing the process of extracting and outputting text in a table by a data extraction program; Explanatory drawing which shows the example of the format which a data extraction program outputs tabular data. FIG. 4 is a flow diagram showing a process of extracting information indicating properties of extracted table data from a text file with coordinate information by a data extraction program. FIG. 10 is a chart showing an example of a text pattern used for extracting information indicating properties of extracted table data from a text file with coordinate information; FIG. 4 is a chart showing an example of sentence patterns used for extracting information indicating properties of extracted graph data from a text file with coordinate information; FIG. 4 is a flow chart showing the processing when the data extraction program executes graph detection and output.

Embodiments of the invention are described below with reference to the drawings. It should be noted that the embodiments described below do not limit the claimed invention, and not all of the elements and combinations thereof described in the embodiments are essential to the solution of the invention. do not have.

In the configuration of the invention described below, the same reference numerals may be used in common for the same parts or parts having similar functions between different drawings, and redundant description may be omitted.

When there are a plurality of elements having the same or similar functions, they may be described with the same reference numerals and different suffixes. However, if there is no need to distinguish between multiple elements, the subscripts may be omitted.

Notations such as “first”, “second”, “third” in this specification etc. are attached to identify the constituent elements, and do not necessarily limit the number, order, or content thereof isn't it. Also, numbers for identifying components are used for each context, and numbers used in one context do not necessarily indicate the same configuration in other contexts. Also, it does not preclude a component identified by a certain number from having the function of a component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings, etc. may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings and the like.

Elements presented herein in the singular shall include the plural unless the context clearly dictates otherwise.

In the following explanation, the processing may be explained with the subject of "program", but the program is executed by a processor (e.g. Central Processing Unit (CPU)) to appropriately perform the specified processing. The subject of processing may be a program because it is performed using storage resources (eg, memory) and/or communication interface devices (eg, ports). A process described using a program as a subject may be a process performed by a processor or a computer having the processor.

The embodiments detailed below extract desired objects such as tables and graphs from digital documents, and are summarized below. In one embodiment, the system converts a PDF file to a text file with coordinate information and detects table captions from the text file with coordinate information. Then, the document page of the PDF file in which the caption is detected is converted into an image file. Next, image analysis is performed on the page converted into the image file to detect a table. At this time, the range of the image for which the image analysis is performed is limited based on the coordinates of the table caption. Coordinates for table captions are obtained from a text file with coordinate information.

Although PDF is used as an example in the embodiment, any digital document containing objects (text, images, etc.) in document pages and coordinate information of the objects can be used. A text file or an image file with coordinate information can be easily generated from such a digital document.

By performing image analysis only on pages that include table captions, it is possible to avoid erroneous detection of a table that is not a table from a page that does not include a table. In addition, by limiting the range of image analysis from the position of the caption of the table, the information that becomes noise in image analysis is reduced, and the possibility of misdetection of a table that is not a table but is recognized as a table is reduced. be able to. Further, by performing table detection by image analysis, information such as ruled lines can be used in table detection, and the range of the table can be accurately recognized. In general, a table is a representation of data such as letters and numbers in cells separated by vertical and horizontal ruled lines in a grid pattern, but some or all of the ruled lines may be omitted. . Also, cells may be partially merged or separated.

Layout recognition is performed on the table detected by image analysis. At this time, for tables that lack image information, such as tables that contain only horizontal ruled lines, text that explains the structure of the table can be extracted from a text file with coordinate information and used to estimate the layout of the table. . This makes it possible to accurately recognize the layout of the table.

Extract the coordinate information of each cell of the table for which detection and layout recognition are performed by image analysis. This coordinate information is collated with the contents of the text file with coordinate information, and the text of each cell is extracted from the text file with coordinate information. This makes it possible to avoid erroneous recognition of text by OCR. Also, a text file with coordinate information and a predefined sentence pattern can be compared to extract the text explaining the contents of the table as information indicating the properties of the table data.

In another embodiment, the PDF file is converted to a text file with coordinate information, and the caption of the graph is detected from the text file with coordinate information. Then, the document page of the PDF file in which the caption is detected is converted into an image file. Next, image analysis is performed on the page converted into the image file to detect the graph. At this time, the range of the image for which the image analysis is performed is limited based on the coordinates of the graph caption. Coordinates for graph captions are obtained from a text file with coordinate information. By performing image analysis only on pages that include graph captions, it is possible to avoid erroneous detection of graphs that are not graphs but are detected as graphs from pages that do not include graphs. In addition, by limiting the range of image analysis from the position of the caption of the graph, the information that becomes noise in image analysis is reduced, and the possibility of misdetection of graphs, such as detecting things that are not graphs as graphs, is reduced. can be reduced. A graph is generally a diagram showing the relationship between two or more mutually related quantities.

Also, the text file with coordinate information and a predefined sentence pattern can be compared, and the text explaining the contents of the graph can be extracted from the text file with coordinate information as information indicating the information of the graph data.

The outline of the computer system of the data extraction device according to the first embodiment will be described with reference to FIG. FIG. 1 shows the hardware configuration of a computer system according to an embodiment. The data extraction device 100 executes a data extraction program to extract tabular data from the PDF file. A CPU 102 is connected to a memory 103 as a main storage device via a bus 101 and executes a data extraction program on the memory 103 .

CPU 102 and memory 103 are connected via bus 101 to disk device 104, which is a secondary storage device. A data extraction program executed on the CPU 102 can read a PDF file stored in the disk device 104 into the memory 103 and write data to the PDF file stored in the disk device 104.例文帳に追加

The above hardware configuration can be configured with a general server. Although not shown, various input devices and output devices that are provided in general servers may be provided. The above hardware configuration may be composed of a single server, or any part of the input device, output device, CPU, and disk device may be composed of other servers connected by a network. . Further, in the present embodiment, functions equivalent to those configured by software can be realized by hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit).

In this embodiment, a PDF file is used as an example of a target file. However, any object (text, image, etc.) in a document page and coordinate information indicating where the object exists in the page are included. For example, it may be a digital document in any other format.

FIG. 2 shows a functional block configuration of a data extraction program 200 according to the first embodiment. A conversion processing unit 201 reads a PDF file stored in the disk device 104 and converts it into a text file with coordinate information. Also, the PDF file stored in the disk device 104 is read and converted into an image file.

A text data analysis unit 202 analyzes a text file with coordinate information and extracts table captions. Also, the text file with coordinate information is analyzed to extract the text describing the structure of the table and the text indicating the properties of the table. Here, the caption refers to a title or title attached to a photograph, illustration, table, graph, etc. In Example 1, the caption attached to the table (for example, "Table 1: Data of ○○○") is extracted. .

The image analysis unit 203 analyzes the image file based on the table caption information received from the text data analysis unit 202 to detect the table. Also, the text data analysis unit 202 analyzes the text file with coordinate information, and based on the text explaining the structure of the table, the table layout is recognized. Then, the table coordinate information is extracted.

The data storage unit 204 collates the table coordinate information extracted by the image analysis unit 203 with the text file with coordinate information, and extracts the text in the table. Then, output the extracted text as a file or DB table.

FIG. 3 shows a process 300 for extracting table captions in a series of processes for extracting table data by the data extraction program 200 according to the first embodiment. First, the conversion processing unit 201 converts a PDF file from which table data is to be extracted into a text file with coordinate information (process 310).

A file 400 in FIG. 4 shows an example of a text file with coordinate information. As shown in FIG. 4, a tag is attached to each document page of the converted PDF file, and the size of the page in the horizontal and vertical directions (page size) and the page number are added as meta information. Also, each text in the document page is tagged, and the coordinate information of the text is added as meta information. In the example of FIG. 4, the horizontal and vertical coordinates of the upper left portion of the text and the width and height of the text are provided as coordinate information. In the example, a PAGE tag is given to each page and a TOKEN tag is given to each text, but the tag names may be different. Conversion to a text file with coordinate information can be executed by using an existing text extraction tool such as PDFMiner, or you can develop your own conversion process and incorporate it.

Next, the text data analysis unit 202 extracts the caption of the table from the text file with coordinate information. First, the text file with coordinate information is searched for text containing keywords such as "table" and "table" that are likely to be included in the caption of the table (process 311). Search is performed on the text given the TOKEN tag. If you find text containing keywords, extract it.

Next, based on the coordinate information in the text file with coordinate information, the text around the text containing the keyword is also extracted (process 312). The range for extracting text is set in advance. Then, the entire extracted text is compared with a preset sentence pattern (process 313).

FIG. 5A is an example of a table storing preset sentence patterns for matching with extracted text. These sentences are expressions used when citing or referring to tables, etc. in the text, and there is little possibility that these sentences will be used as captions for tables. A sentence pattern as shown in the table 500 is defined in advance and stored in the disk device 104 or the like.

If the entire text is a series of sentences that match any of the sentence patterns included in the table 500, such as "Table XX indicates XX" (XX and XX are arbitrary words/characters). If so, it is determined that the text containing the keyword is not a table caption (process 314: No).

If it is not a series of sentences, the text containing the keyword is judged to be a table caption (process 314: Yes), the page number and page size of the page containing the text, the text, the coordinates of the text The information is extracted as table caption information and sent to the image analysis unit 203 (process 315). A series of processes from process 312 to process 315 are repeated by the number of texts obtained by keyword search (process 316).

A table 510 in FIG. 5B shows an example of table caption information sent to the image analysis unit 203 . The page number and page size are recorded for each page, and the detected caption and its coordinates (coordinates on the text file with coordinate information) are recorded for each page.

FIG. 6 shows a process 600 in which the data extraction program 200 performs table detection and layout recognition in a series of processes for extracting table data. The table detection and layout recognition process 600 is performed for each page on which a table caption was detected in process 300 .

First, the image analysis unit 203 refers to the table 510 created by the text data analysis unit 202 in process 300, and confirms the page number of the page including the table caption (process 610). Then, the document page of the PDF file is converted into an image file (process 611). Conversion to an image file can be executed by using an existing image conversion tool such as pdftoppm (https://www.xpdfreader.com/pdftoppm-man.html), or a unique conversion process can be developed and incorporated.

Next, the table caption coordinate information (see table 510 in FIG. 5B) extracted by the text data analysis unit in process 300 is adjusted to the coordinate scale of the image file (process 612). Based on the size of the relevant document page on the image file, the coordinate information of the table caption is converted from the coordinate information on the text file with coordinate information to the coordinate information on the image file. The conversion can be performed with the following formula.

Table caption abscissa (image file) =
Abscissa of table caption (text file) x (horizontal size of document page (image file) / horizontal size of document page (text file))

Table caption ordinate (image file) =
Vertical coordinate of table caption (text file) x (Vertical size of document page (image file) / Vertical size of document page (text file))

The size of the document page on the image file can be obtained by image processing libraries such as OpenCV (https://opencv.org).

Next, the image analysis unit 203 performs processing for limiting the image range for performing table detection (processing 613). The image range is limited based on the coordinates of the table caption. In the flow of FIG. 6, only the page from which the caption is extracted is image-converted, but all pages may be image-converted in advance.

FIG. 7 shows an example in which the image analysis unit 203 limits the image range for performing table detection. In image 700, the table detection is limited to area 701 on the premise that the table caption is located at the top left of the table. In the image 710, the table caption is limited to areas 711 and 712 on the premise that it is arranged at the top left of the table.

As shown in the image 710, when multiple table captions are included in the same page, the image range is adjusted so that the image range for which table detection is performed does not include multiple tables, due to the positional relationship between the table captions. limit.

The example assumes that the table caption is in the upper left corner of the table, but the position of the table caption varies from document to document. When limiting the image range for table detection, it is necessary to consider the positional relationship between the table caption and the table and change the processing for limiting the image range.

In this case, for example, the user may specify the positional relationship between the table caption and the image range for which table detection is to be performed before the process 600 starts. Alternatively, the process 613 may be omitted and the entire page including the table caption may be the image range for table detection. However, using the positional relationship between the table caption and the image range in which table detection is performed can be expected to reduce the amount of processing and improve accuracy.

Next, the image analysis unit 203 executes table detection from the image range limited in process 613 (process 614). Any method may be used for table detection. For example, there is a method of detecting rectangles using an image processing library or the like and detecting the detected rectangles as a table. In addition, straight lines are detected using an image processing library, etc., and the highest and lowest straight lines in the vertical coordinate are taken as the upper and lower sides of the table, and the range surrounded by the upper and lower sides may be detected as a table.

Next, the image analysis unit 203 executes layout recognition of the table detected in process 614 (process 615). This process is performed for each table identified in process 614 . Any method may be used for table layout recognition.

FIG. 8A is a flowchart of an example of table layout recognition processing. For example, process 615A in FIG. 8A shows an example of layout recognition of a table in which each cell range of the table is represented by vertical and horizontal ruled lines.

In this example, first, using an image processing library or the like, rectangle detection processing is executed for the range of the table detected in processing 614 (processing 8110). Then, each detected rectangle is recognized as each cell of the table. Then, the coordinate information of each cell, for example, the abscissa and ordinate of the upper left point of the table, and the cell width and height are extracted using an image processing library or the like (process 8111). Then, based on this coordinate information, cells adjacent to each other in the vertical and horizontal directions are determined for each cell, and cells belonging to the same column and the same row are recognized (process 8112). The above processing is executed by the image analysis unit 203 through image analysis.

Process 615B in FIG. 8B shows an example of layout recognition for a table containing only horizontal ruled lines.
FIG. 8C is a flow diagram showing details of operation 8212 of operation 615B.
In the example of process 615B, the image analysis unit 203 first uses an image processing library or the like to perform histogram analysis on the range of the table detected in process 614 (process 8210). A histogram is created by counting the number of colored pixels in the vertical and horizontal directions of the table image. Then, each cell of the table is recognized using the trough portion of the histogram as the boundary of the cell.

9A and 9B show an example of recognizing cells by histogram analysis in a table containing only horizontal ruled lines. A table 9110 in FIG. 9A shows an example of a table for histogram analysis, and a table 9120 in FIG. 9B shows the results of histogram analysis on the table 9110. FIG. In the table 9120, the area surrounded by the dotted line is recognized as a cell, and the coordinate information of each cell, for example, the abscissa and ordinate of the upper left point in the table, the width and height of the cell, and the image processing library etc. (process 8211).

Next, cells belonging to the same column and same row are recognized (process 8212). FIG. 8C shows details of processing 8212 for recognizing cells belonging to the same column and same row in a table that includes only horizontal ruled lines. First, for each cell, the coordinate distance to other cells is calculated (process 8310). Cells that are closer in the vertical direction are placed in the same column, and cells that are closest in the horizontal direction are placed in the same row (process 8311). However, this may result in misrecognition of rows and columns.

Table 9210 in FIG. 9C shows a case where the columns of table 9110 are correctly recognized. Columns 9213 and 9214 connect to merged cell 9217 and are recognized as the same column 9211 . Similarly, columns 9215 and 9216 connect to merged cell 9218 and are recognized as the same column 9212 .

Table 9220 of FIG. 9D illustrates the case where columns of table 9110 are misrecognized. Originally, column 9224 should be connected to merged cell 9227 and belong to column 9221 like column 9223 . However, the column 9224 is connected to the columns 9225 and 9226 by the merged cell 9228 and is recognized as the same column 9222 . This is because the cell closest in coordinate distance to the cell 9229 has been determined to be the merged cell 9228 .

In order to correct such erroneous recognition of columns and rows, in this embodiment, the data storage unit 204 performs a process of estimating the cell layout from the text of the PDF document. First, a process of extracting text in each cell from a text file with coordinate information is performed (process 8312). In order to extract the text in each cell from the text file with coordinate information, the image analysis unit 203 converts the coordinate information of each cell from the coordinate information on the image file to the coordinate information on the text file with coordinate information. Convert. Conversion is performed by the following formula.

Cell abscissa (text file) =
Horizontal coordinate of cell (image file) x (horizontal size of document page (text file) / horizontal size of document page (image file))

Cell ordinate (text file) =
Vertical coordinate of cell (image file) x (Vertical size of document page (text file) ÷ Vertical size of document page (image file))

cell width (text file) =
Cell width (image file) x (horizontal size of document page (text file) / horizontal size of document page (image file))

cell height (text file) =
Cell height (image file) x (horizontal size of document page (text file) / horizontal size of document page (image file))

The data storage unit 204 compares this coordinate information with the coordinate information in the text file with coordinate information, and extracts the text existing within the range of the cell on the coordinates as the text in the cell. Taking the file 400 as an example, the coordinate information of each text attached with the TOKEN tag is referred to, and the center coordinates of the text are obtained by the following formula.

Center coordinates (X, Y) = (abscissa + (width/2), ordinate + (height/2))

If the center coordinates are within the coordinate range of the cell, the text is determined to be the text in the cell.

Next, the text data analysis unit 202 extracts and refers to the description of the table from the text file with the coordinate information (process 8313), and corrects the rows and columns (process 8314). First, extract all texts with TOKEN tags in a text file with coordinate information, and form sentences based on the coordinate information of each text. This is done by recognizing text that is continuous on coordinates as a series of sentences. Examine whether or not the sentence thus formed contains a sentence pattern explaining the structure of the table, and if so, determine a sentence containing the sentence pattern as an explanatory sentence of the table, Extract.

FIG. 9E shows an example sentence pattern 9300 that describes the structure of a table. The sentence pattern 9300 is stored in advance in the disk device 104 or the like. In the example shown in the figure, there is one sentence pattern, but a plurality of sentence patterns may be prepared in advance, and if any sentence pattern is included, the sentence may be determined as the explanatory sentence of the table.

Next, the explanation of the table is analyzed. A descriptive text 9310 in FIG. 9E shows an example in which the descriptive text of the table 9110 is extracted by the text pattern 9300 . In the sentence pattern 9300, a cell containing "XX" in the cell text and a cell containing "△△" in the cell text are connected by a merged cell containing "○○" in the cell text, and are in the same column. to form Therefore, by referring to the descriptive text 9310, a merged cell containing "Step" in the cell text is connected to a cell containing "Compound" in the cell text and a cell containing "Yield" in the cell text. I understand.

Since the text corresponding to each cell has been extracted in process 8312, the connection relationship of the cells can be estimated based on the description 9310 by referring to the extracted in-cell text of each cell. For example, in row/column recognition based on coordinate distances between cells by the image analysis unit, as shown in table 9220 in FIG. Even if , it can be found that it is inconsistent with the connection relation of the cell based on the description 9310 by comparing the description 9310 with the text in the cell. Thus, as shown in table 9210 of FIG. 9C, the correct column recognition can be corrected, with combined cell 9217 and cell 9119 forming the same column 9211. FIG.

After completion of the table layout recognition process, the coordinate information of the table is output (process 616).

Coordinate information 1000 in FIG. 10 shows an output example of table coordinate information whose layout is recognized by the table layout recognition processing 515 . A unique ID is assigned to each cell constituting the table, and coordinate information of each cell (in the example of coordinate information 1000, the abscissa, ordinate, cell width, and cell height of the upper left corner of the cell) is output. This coordinate information is coordinate information on the image file. Also, for each cell, the ID of the adjacent cell recognized by the table layout recognition process is output. Adjacent cells are those with the closest horizontal coordinate distance among the cells recognized as being on the same row, and those with the closest horizontal coordinate distance among the cells recognized as being on the same column. Adjacent cells on the same row exist on the left and right, and adjacent cells on the same column exist on the top and bottom. If the cell is a merged cell, it may be adjacent to multiple cells on the left, right, top and bottom. Further, the extracted coordinate information 1000 of the table is provided with the corresponding caption in the table 510 . The coordinate information 1000 is sent from the image analysis section 203 to the data storage section 204 .

After executing the processes 615 and 616 for all the tables detected in the process 614, the table layout recognition process is terminated (process 617: Yes). Otherwise, processing 615 and processing 616 are subsequently performed (processing 617: No).

After the processes 610 to 617 have been executed for all pages in which table captions have been detected, the table detection/layout recognition process ends (process 618: Yes). Otherwise, processing 610 to processing 617 are continuously performed (processing 618: No).

FIG. 11 shows a process of extracting and outputting text in a table in a series of processes for extracting table data by the data extraction program 200 according to the first embodiment. This process is performed as many times as the number of tables detected in process 614 .

Based on the table coordinate information 1000 received from the image analysis unit 203, the data storage unit 204 extracts the text in the cell from the text file with the coordinate information for each cell (process 1110). The text extraction method is the same as the method shown in process 8312 .

Next, the extracted text is shaped into data that reproduces the structure of the original table, and is output (process 1111). Based on the table coordinate information received from the image analysis unit 203, the adjacent cells of each cell are recognized, the original table structure is reproduced, and the data is output. The output format can be HTML table representation or DB table structure.

The HTML file 1210 in FIG. 12 expresses the table 1200 in HTML table representation. As shown in the HTML file 1210, each cell is given a th tag or a td tag. Then, the cells in the same row are enclosed in tr tags, and the entire cell is enclosed in table tags to express the structure of the table. As shown in the HTML file 1210, a colspan attribute is added to represent horizontally merged cells, and a rowspan attribute is added to represent column-wise merged cells.

By adding a th or td tag to the text of each cell extracted in process 1110, referring to adjacent cells of each cell recorded in table coordinate information 1000, and enclosing cells in the same row with tr tags , can output the extracted tabular data in HTML table representation. Add the colspan attribute to merged cells that have multiple adjacent cells in the column direction, and add the rowspan attribute to merged cells that have multiple adjacent cells in the row direction.

The output format does not necessarily have to be an HTML table representation, and as already mentioned, a DB table structure or the like may be used. When outputting, append the caption of the table as metadata. When outputting as an HTML table expression, a method such as adding a caption of the table as attribute information of the table tag can be considered. In the case of DB tables, a method such as setting the caption of the table to the table name can be considered. The output data is saved on the disk device 104 in the form of a file, DB, or the like.

FIG. 13 shows a process in which the data extraction program 200 according to the first embodiment extracts and outputs information indicating properties of table data in a series of processes for extracting table data.

First, the text data analysis unit 202 extracts all texts to which the TOKEN tag is attached in the text file with coordinate information, and forms sentences based on the coordinate information of each text. This is done by recognizing text that is continuous on coordinates as a series of sentences. It is checked whether the text thus formed contains a text pattern that explains the properties of the table (process 1310).

If it is included, the sentence including the sentence pattern is extracted as information indicating the properties of the table and transferred to the data storage unit. The data storage unit associates the passed information (text) with the table data output in FIG. 11 and saves it in the form of a file, DB, or the like on the disk device 104 (process 1311).

FIG. 14 is a table diagram showing a sentence pattern 1400, which is an example of a sentence pattern for extracting a text explaining the properties of a table.

By comparing the table caption, which is the metadata of the table data output in the process 1111, with the information extracted in the process 1311, the properties of the table data output in the process 1111 can be identified by the application executing the data analysis at the time of data analysis. , can be known.

In Example 1, an example of extracting a table in a digital document was shown. Objects to be extracted are not limited to tables, and may be graphs, drawings, or the like. A second embodiment describes an example of extracting a graph from a digital document.

An overview of the computer system of the data extraction device according to the second embodiment will be described. The hardware configuration of the computer system according to the second embodiment may be the same as in FIG. Since the second embodiment is an example of extracting graph data from a PDF file, the data extraction device 100 executes a data extraction program for graph extraction and extracts graph data from the PDF file.

The functional block configuration of the data extraction program 200 stored in the memory 103 for graph extraction may be the same as in FIG. However, the specific processing executed by each functional block 201 to 204 is partly different from the first embodiment. Description is made below with reference to FIG.

A conversion processing unit 201 reads a PDF file stored in the disk device 104 and converts it into a text file with coordinate information. Also, the PDF file stored in the disk device 104 is read and converted into an image file. This treatment may be the same as in the first embodiment.

A text data analysis unit 202 analyzes a text file with coordinate information and extracts graph captions. It also analyzes the text file with the coordinate information and extracts the text that indicates the properties of the graph.

The process of extracting graph captions is the same as the flow shown in FIG. 3 of the first embodiment. Search for text containing keywords that are likely to be included in . Also, in the processing 313 for matching with a sentence pattern, a sentence pattern for extracting a graph is used instead of the sentence pattern for extracting the table shown in FIG. 5A. A specific example is a sentence pattern that describes the contents of a "graph" or "graph".

If the text around the extracted keyword does not form a sentence pattern, it is determined that the text containing the keyword is a graph caption, and the same information as in FIG. replaced) is sent to the image analysis unit 203 (process 315). A series of processes from process 312 to process 315 are repeated by the number of texts obtained by keyword search (process 316).

Also, the processing for extracting the text indicating the characteristics of the graph is similar to that shown in FIG. Use sentence patterns to extract.

FIG. 15 is a table showing a sentence pattern 2300, which is an example of a sentence pattern for extracting a text that indicates the properties of a graph.

The image analysis unit 203 analyzes the image file based on the graph caption information received from the text data analysis unit 202 to detect the graph.

A data storage unit 204 outputs the graph extracted by the image analysis unit 203 in the form of a file or the like.

FIG. 16 shows a process 2000 for detecting and outputting a graph in a series of processes for extracting graph data by the data extraction program 200 . The graph detection/output process 2000 is executed for each page where the graph caption was detected in process 300 .

First, the image analysis unit 203 refers to the table 510 (however, table captions are replaced with graph captions) created by the text data analysis unit 202 in process 300, and confirms the page numbers of pages including graph captions (process 2010). ). Then, the document page of the PDF file is converted into an image file (process 2011). Conversion to an image file is the same as in the first embodiment.

Next, the coordinate information of the graph caption extracted by the text data analysis unit 202 in the process 300 is adjusted to the coordinate scale of the image file (process 2012). Based on the size of the document page on the image file, the coordinate information of the graph caption is converted from the coordinate information on the text file with coordinate information to the coordinate information on the image file. Size conversion can be performed in the same manner as the table caption by replacing "table caption" with "graph caption" in the conversion formula of the first embodiment.

Next, the image analysis unit 203 performs processing for limiting the image range for executing graph detection (processing 2013). The image range is limited based on the coordinates of the graph caption. The method of limiting the image range in which the image analysis unit 203 executes graph detection is the same as in the first embodiment.

Next, the image analysis unit 203 executes graph detection from the image range limited in process 2013 (process 2014). Any method may be used for graph detection. For example, there is a method of detecting rectangles using an image processing library or the like and detecting the detected rectangles as a graph. Further, straight line detection may be performed using an image processing library or the like, two orthogonal straight lines may be recognized as the vertical axis and the horizontal axis of the graph, and the range surrounded by both straight lines may be detected as a graph. After detecting the graph, the image analysis unit 203 sends the image of the detected graph to the data storage unit 204 .

The data storage unit 204 that has received the graph image stores it in the disk device 104 in the form of a file or the like. At the time of output, the caption of the graph is given as metadata. Any method can be used to add the metadata, but one possible method is to use the caption of the graph as the file name.

In the second embodiment, as in FIG. 13 of the first embodiment, the data extraction program 200 can extract and output information indicating the properties of graph data in a series of processes for extracting graph data. can.

First, the text data analysis unit 202 extracts all texts to which the TOKEN tag is attached in the text file with coordinate information, and forms sentences based on the coordinate information of each text. This is done by recognizing text that is continuous on coordinates as a series of sentences. It is checked whether or not the sentence thus formed contains a sentence pattern (see FIG. 15) explaining the properties of the graph (process 1310). , is extracted as information indicating the properties of the graph and passed to the data storage unit 204 . The data storage unit 204 saves the transferred information (text) on the disk device 104 in the form of a file, DB, or the like (process 1311).

By comparing the graph caption, which is the metadata of the graph data output in the process 2015, with the information extracted in the process 2211, the properties of the graph data output in the process 2015 can be identified by the application executing the data analysis. , can be known.

According to the embodiments described in detail above, by using both image information and text with coordinate information, it is possible to accurately extract table data and graph data from, for example, a PDF file. In addition to table data and graph data, it is possible to extract information indicating the properties of the table and graph. These will improve the quality of data analysis.

The data extraction method described in this specification can be applied not only to PDF files, but also to digital documents in general, which are composed of objects in document pages and coordinate information of the objects. The techniques described in the examples improve the quality of data analysis for these digital documents.

Data extraction device 100, bus 101, CPU 102, memory 103, disk device 104, data extraction program 200, conversion processing unit 201, text data analysis unit 202, image analysis unit 203, data storage unit 204

Claims (8)

a conversion processing unit that converts a digital document containing an object in a document page and coordinate information of the object into image data and text data with coordinate information;
a text data analysis unit that extracts captions associated with the object from the text data;
an image analysis unit that detects the object based on the caption information extracted from the image data;
The conversion processing unit is
Converting the page from which the caption was extracted from the digital document into image data;
The image analysis unit
performing the object detection on the image data of the page from which the caption was extracted ;
Data extraction system from digital documents. a conversion processing unit that converts a digital document containing an object in a document page and coordinate information of the object into image data and text data with coordinate information;
a text data analysis unit that extracts captions associated with the object from the text data;
an image analysis unit that detects the object based on the caption information extracted from the image data;
The conversion processing unit is
Converting the page from which the caption was extracted from the digital document into image data;
The image analysis unit
Limiting an image range in which image analysis is performed on the image data of the page from which the caption is extracted based on the coordinate information of the caption, and detecting the object ;
Data extraction system from digital documents. a conversion processing unit that converts a digital document containing an object in a document page and coordinate information of the object into image data and text data with coordinate information;
a text data analysis unit that extracts captions associated with the object from the text data;
an image analysis unit that detects the object based on the caption information extracted from the image data;
The object is a table,
A data storage unit that compares the detected coordinate information of the table with the text data with the coordinate information and extracts the text in the table,
The image analysis unit
Extract each cell of the detected table,
The data storage unit
matching the coordinate information of each cell with the text data with the coordinate information, and extracting the text in each cell ;
Data extraction system from digital documents. The text data analysis unit
extracting text explaining the structure of the table by matching the text data with a preset sentence pattern;
The data storage unit
recognizing the layout of a table based on text in said cells and text describing the structure of said table;
4. A system for extracting data from digital documents according to claim 3 . The text data analysis unit
Searching text that is a candidate for a caption by matching the text data with a keyword, and identifying whether the candidate for the caption is a caption or a part of the text by matching the candidate for the caption with a preset sentence pattern;
A system for extracting data from digital documents according to any one of claims 1 to 3 . the object is tabular data or graph data;
A system for extracting data from digital documents according to any one of claims 1 to 3 . The text data analysis unit
extracting information indicating the properties of the object from the text data with the coordinate information based on a sentence pattern defined in advance ;
A system for extracting data from digital documents according to any one of claims 1 to 3 . The caption information is at least one of the page where the caption is detected and the coordinates where the caption is detected,
A system for extracting data from digital documents according to any one of claims 1 to 3 .

Images