JP2020170445A - Data extraction method and system from digital document - Google Patents

Abstract

To improve the quality of data analysis for digital documents.SOLUTION: A data extraction system from digital documents in one preferable aspect of the present invention includes: a conversion processing unit that converts a digital document including an object in a document page and coordinate information on the object into text data with image data and the coordinate information; a text data analysis unit that extracts a caption related to an object a user wants to extract from the text data; and an image analysis unit that detects the object the user wants to extract from the image data based on information on the extracted caption.SELECTED DRAWING: Figure 3

Description

The present invention relates to a method for extracting data such as graphs and tables from a digital document.

A service that analyzes data such as newspaper articles, papers, and corporate financial statements published as digital documents by data analysis using artificial intelligence (AI), etc., and provides valuable information to users. It has become. As a preparation for data analysis, it is necessary to convert a large amount of digital documents collected from the Web etc. into a format that is easy to use. Digital documents come in a variety of file formats, such as Hyper Text Markup Language (HTML) and Extensible Markup Language (XML), and digital documents in these formats show the structure of the data as well as the data itself. Contains meta information. By analyzing the structure of such digital documents, only the data to be analyzed, such as the title of the digital document, text, author, publication date, table, and graph, is extracted and stored in a storage device such as Data Base (DB). It is necessary to perform the conversion process to be performed as data preparation for data analysis.

Currently, many digital documents are distributed as Portable Document Format (PDF) files, and analysis of PDF files is required for data analysis. When performing data analysis, it is necessary to analyze the structure of the PDF file and extract the data to be analyzed as data preparation. Data such as tables and graphs are likely to store important information, and extraction of table and graph data is strongly required.

PDF files have a complicated format such as a mixture of text and binary, and it is difficult to analyze the data structure as it is. In addition, the PDF file is composed of objects (text, images, etc.) in the document page and coordinate information indicating where the object exists in the page, and indicates what kind of data each object is. The information is not embedded. For example, information such as whether the text in the document page is the body of the document or the table data is not embedded in the PDF file, and in addition to this, it is a complicated format, and in the PDF file, Difficult to identify and extract data in documents.

As a conventional technique, there is a method in which text data of a PDF file is converted into a format in which the data structure can be easily analyzed, and then table data is extracted. Extraction of table data from PDF files is done by table discovery, layout recognition, and extraction of text in the table. Table detection is a process of detecting a table from a document page of a PDF file, and layout recognition is a process of recognizing which row and column each cell of the table belongs to. Extracting the text in the table is a process of extracting the detected text in the table. The extracted text is saved in the storage device in the form of an HTML table representation that reproduces the structure (layout) of the original table or a DB table.

One of the conventional techniques is a method of converting a PDF file into an image file and analyzing the data structure. It analyzes image information such as table ruled lines (table ruled lines, etc.), detects the table, and recognizes the table layout. The text in the table is extracted by OCR (Optical Character Recognition / Reader). Patent Document 1 discloses a technique of extracting a table as image data and extracting the table data by image analysis.

The other is a method of converting a PDF file into a text file with coordinate information and analyzing the data structure. If you use a text extraction tool such as PDFMiner (https://github.com/euske/pdfminer/), you can obtain a text file with the coordinates in the page added to the text in the PDF document. Coordinates are given in units that are organized to some extent (units, sentences, etc.). Table data is detected and layout recognition is performed by analyzing the text with coordinate information and analyzing the positional relationship between the texts. As an example, Non-Patent Document 1 and the like can be mentioned.

US20150093021A1

Martha et.al., “TAO: System for Table Detection and Extraction from PDF Documents”, FLAIRS Conference, 2016

The method of extracting table data by image analysis of Patent Document 1 has a problem that an object other than a table on a page is erroneously detected as a table. Further, in table layout recognition, there is a problem that the layout of the table cannot be accurately recognized in a table lacking image information such as a table containing only horizontal ruled lines. In addition, there is a problem that OCR text misrecognition occurs in the extraction of text in the table.

On the other hand, in the method of analyzing the text with coordinate information of Non-Patent Document 1 and extracting the table data, the image information such as the ruled lines of the table cannot be used, so that the range of the table is correctly recognized in the detection of the table. But there is a difficult problem.

Therefore, an object of the present invention is to improve the quality of data analysis for digital documents.

A preferred aspect of the present invention is a conversion processing unit that converts an object in a document page and a digital document containing the coordinate information of the object into text data with image data and coordinate information, and an object to be extracted from the text data. It is a data extraction system from a digital document including a text data analysis unit that extracts related captions and an image analysis unit that detects an object to be extracted based on the information of the extracted captions from the image data.

Another preferred aspect of the present invention is a method of extracting data from a digital document, which extracts a desired object from the digital document, generating text data with coordinate information from the digital document, text data generation processing, keywords. From a digital document, including a caption detection process that detects a caption of a desired object from text data by a search, and an object extraction process that extracts a desired object from the image data of a page of a digital document in which a caption is detected. Data extraction method.

Improves the quality of data analysis for digital documents. More specific issues, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

The block diagram which shows the hardware configuration of the data extraction apparatus which concerns on Example. The logical block diagram which shows the logical structure of the data extraction program which concerns on Example. A flow chart showing the processing when detecting the caption of the table. A table diagram showing an example of a text file with coordinate information when a PDF file is converted into a text file with coordinate information. A chart showing an example of a sentence pattern used to distinguish between a table caption and the text when detecting a table caption. The figure which shows the example of the detection result of the caption of the table sent to the image analysis part. A flow chart showing the processing when table detection and layout recognition are performed. Explanatory diagram showing how to limit the range of image analysis when performing table detection. A flow diagram showing the processing when performing layout recognition of a table in which cells are represented in a grid pattern by ruled lines. A flow diagram showing processing when executing layout recognition of a table lacking ruled line information. A flow chart showing processing when performing row / column recognition of a table that includes merged cells and lacks ruled line information. The figure which shows the example of the table which performs the histogram analysis. The figure which shows the example of the result of recognizing a cell by the histogram analysis. An explanatory diagram showing an example when the layout is correctly recognized in the layout recognition of the table. Explanatory drawing which shows an example when the layout is erroneously recognized in the layout recognition of a table. An explanatory diagram showing a method of analyzing the body of a PDF document existing in a coordinate information text file and recognizing the layout of a table. A table diagram showing the structure of the coordinate information of the table extracted by the data extraction program. A flow chart showing the process by which the data extraction program extracts and outputs the text in the table. An explanatory diagram showing an example of a format in which a data extraction program outputs table data. A flow chart showing a process in which a data extraction program extracts information indicating the properties of the extracted table data from a text file with coordinate information. A table diagram showing an example of a sentence pattern used for extracting information indicating the properties of the extracted table data from a text file with coordinate information. A table diagram showing an example of a sentence pattern used for extracting information indicating the properties of the extracted graph data from a text file with coordinate information. A flow diagram showing the processing when a data extraction program detects and outputs a graph.

Hereinafter, examples of the present invention will be described with reference to the drawings. It should be noted that the examples described below do not limit the inventions claimed in the claims, and not all of the elements and combinations thereof described in the examples are essential for the means for solving the invention. Absent.

In the configuration of the invention described below, the same reference numerals may be used in common among different drawings for the same parts or parts having similar functions, and duplicate description may be omitted.

When there are a plurality of elements having the same or similar functions, they may be described by adding different subscripts to the same reference numerals. However, if it is not necessary to distinguish between a plurality of elements, the subscript may be omitted for explanation.

Notations such as "first", "second", and "third" in the present specification and the like are attached to identify the constituent elements, and do not necessarily limit the number, order, or contents thereof. is not. Further, the numbers for identifying the components are used for each context, and the numbers used in one context do not always indicate the same composition in the other contexts. Further, it does not prevent the component identified by a certain number from having the function of the component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings and the like.

The components represented in the singular form herein shall include the plural form unless explicitly indicated in the context.

In the following description, processing may be described with "program" as the subject, but the program is executed by a processor (for example, Central Processing Unit (CPU)) to appropriately perform the specified processing. Since it is performed using a storage resource (for example, memory) and / or a communication interface device (for example, a port), the subject of processing may be a program. The process described with the program as the subject may be a process performed by a processor or a computer having the processor.

The embodiments described in detail below are for extracting desired objects such as tables and graphs from a digital document, the outline of which is described below. In the system of one embodiment, the PDF file is converted into a text file with coordinate information, and the caption of the table is detected from the text file with coordinate information. Then, the document page of the PDF file in which the caption is detected is converted into an image file. Next, the page converted into an image file is subjected to image analysis to detect a table. At this time, the range of the image to be analyzed is limited based on the coordinates of the table caption. The coordinates of the table caption are obtained from a text file with coordinate information.

In the embodiment, PDF is used as an example, but the present invention is not limited to this as long as it is an object (text, image, etc.) in the document page and a digital document containing the coordinate information of the object. From such a digital document, a text file or an image file with coordinate information can be easily generated.

By performing image analysis only on pages that include table captions, it is possible to avoid erroneous detection of tables that do not include tables and detect non-tables as tables. In addition, by limiting the range of image analysis from the position of the caption of the table, information that causes noise in image analysis is reduced, and the possibility of erroneous detection of the table that recognizes something other than the table as a table is reduced. be able to. Further, by detecting the table by image analysis, information such as ruled lines can be used in the detection of the table, and the range of the table can be accurately recognized. A table is generally represented by describing data such as characters and numbers in cells divided by vertical and horizontal ruled lines in a grid pattern, but some or all of the ruled lines may be omitted. .. Also, cells may be partially merged or separated.

Layout recognition is performed on the table detected by image analysis. At this time, for a table that lacks image information, such as a table that contains only horizontal ruled lines, the text that explains the structure of the table can be extracted from the text file with coordinate information and used to estimate the layout of the table. .. This makes it possible to accurately recognize the layout of the table.

Extract the coordinate information of each cell of the table for which detection and layout recognition have been executed by image analysis. This coordinate information is collated with the contents of the text file with coordinate information, and the text of each cell is extracted from the text file with coordinate information. This makes it possible to avoid erroneous recognition of text by OCR. Further, the text file with coordinate information can be collated with a predetermined sentence pattern, and the text explaining the contents of the table can be extracted as information indicating the nature of the table data.

In another embodiment, the PDF file is converted into a text file with coordinate information, and the caption of the graph is detected from the text file with coordinate information. Then, the document page of the PDF file in which the caption is detected is converted into an image file. Next, image analysis is performed on the page converted into an image file, and a graph is detected. At this time, the range of the image to be analyzed is limited based on the coordinates of the graph caption. The coordinates of the graph caption are obtained from a text file with coordinate information. By performing image analysis only on pages that include graph captions, it is possible to avoid false detection of graphs that do not include graphs and detect non-graphs as graphs. In addition, by limiting the range of image analysis from the position of the caption of the graph, information that causes noise in image analysis can be reduced, and non-graphs can be detected as graphs, which may lead to false detection of graphs. It can be reduced. In general, a graph is a diagram showing the relationship between two or more quantities that are related to each other.

Further, the text file with coordinate information and the predetermined sentence pattern can be collated, and the text explaining the contents of the graph can be extracted from the text file with coordinate information as information indicating the information of the graph data.

An outline of the computer system of the data extraction device according to the first embodiment will be described with reference to FIG. FIG. 1 shows a hardware configuration of a computer system according to an embodiment. The data extraction device 100 executes a data extraction program and extracts table data from the PDF file. The CPU 102 is connected to the memory 103, which is the main storage device, via the bus 101, and executes a data extraction program on the memory 103.

The CPU 102 and the memory 103 are connected to the disk device 104, which is a secondary storage device, via the bus 101. The data extraction program executed on the CPU 102 can read the PDF file stored in the disk device 104 into the memory 103 and write the data to the PDF file stored in the disk device 104.

The above hardware configuration can be configured with a general server. Although not shown, various input devices and output devices provided in a general server may be provided. The above hardware configuration may be configured by a single server, or any part of the input device, output device, CPU, and disk device may be configured by another server connected by a network. .. Further, in this embodiment, the same function as the function configured by the software can be realized by the hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit).

In this embodiment, a PDF file is taken as an example of the target file, but an object (text, image, etc.) in the document page and coordinate information indicating where the object exists in the page are included. For example, it may be a digital document in another format.

FIG. 2 shows a functional block configuration of the data extraction program 200 according to the first embodiment. The conversion processing unit 201 reads the PDF file stored in the disk device 104 and converts it into a text file with coordinate information. In addition, the PDF file stored in the disk device 104 is read and converted into an image file.

The text data analysis unit 202 analyzes the text file with the coordinate information and extracts the table caption. In addition, a text file with coordinate information is analyzed to extract a text explaining the structure of the table and a text showing the properties of the table. Here, the caption refers to a title or title attached to a photograph, a plate, a table, a graph, or the like, and in Example 1, the caption attached to the table (for example, "Table 1: Data of ○○○") is extracted. ..

The image analysis unit 203 analyzes the image file based on the information of the table caption received from the text data analysis unit 202, and detects the table. Further, the text data analysis unit 202 executes table layout recognition based on the text explaining the structure of the table obtained by analyzing the text file with coordinate information. Then, the coordinate information of the table is extracted.

The data storage unit 204 collates the coordinate information of the table extracted by the image analysis unit 203 with the text file with the coordinate information, and extracts the text in the table. Then, the extracted text is output as a file or DB table.

FIG. 3 shows a process 300 for extracting a table caption in a series of processes in which the data extraction program 200 according to the first embodiment extracts table data. First, the conversion processing unit 201 converts the PDF file from which the table data is extracted into a text file with coordinate information (process 310).

File 400 in FIG. 4 shows an example of a text file with coordinate information. As shown in FIG. 4, a tag is added to each document page of the converted PDF file, and the horizontal / vertical size (page size) of the page and the page number are added as meta information. In addition, a tag is added to each text in the document page, and the coordinate information of the text is added as meta information. In the example of FIG. 4, the abscissa and ordinate coordinates of the upper left part of the text and the width and height of the text are given as coordinate information. In the example, a PAGE tag is added to each page and a TOKEN tag is added to each text, but the tag name may be different. The conversion to a text file with coordinate information can be executed by using an existing text extraction tool such as PDFMiner, or an original conversion process may be developed and incorporated.

Next, the text data analysis unit 202 extracts the caption of the table from the text file with the coordinate information. First, a process of searching a text file with coordinate information for texts including keywords such as "table" and "Table" that are likely to be included in the caption of the table is performed (process 311). The search is performed on the text with the TOKEN tag. If it finds text that contains a keyword, it extracts it.

Next, based on the coordinate information of the text file with coordinate information, the text around the text including the keyword is also extracted (process 312). The range for extracting text is set in advance. Then, the entire extracted text is collated with a preset sentence pattern (process 313).

FIG. 5A is an example of a table in which preset sentence patterns are stored in order to collate with the extracted text. These sentences are expressions used when quoting or referring to a table or the like in the text, and it is unlikely that these sentences will be used as captions for the table. The sentence pattern as shown in the table 500 is defined in advance and stored in the disk device 104 or the like.

If the entire text is a series of sentences that match any of the sentence patterns included in the table 500, such as "Table XX indicates XX" (XX, XX are arbitrary words / characters). Then, it is determined that the text including the keyword is not a table caption (process 314: No).

If it is not a series of sentences, it is determined that the text containing the keyword is a table caption (process 314: Yes), the page number and page size of the page containing the text, the text, and the coordinates of the text. The information is extracted as table caption information and sent to the image analysis unit 203 (process 315). The series of processes from process 312 to process 315 is repeated by the number of texts obtained by the keyword search (process 316).

Table 510 in FIG. 5B shows an example of table caption information sent to the image analysis unit 203. Record the page number and page size for each page, and record the detected caption and its coordinates (coordinates on the text file with coordinate information) for each page.

FIG. 6 shows a process 600 in which the data extraction program 200 performs table detection and layout recognition in a series of processes for extracting table data. The table detection / layout recognition process 600 is executed for each page for which the table caption is detected in the process 300.

First, the image analysis unit 203 refers to the table 510 created in the process 300 by the text data analysis unit 202, and confirms the page number of the page including the table caption (process 610). Then, the document page of the PDF file is converted into an image file (process 611). Conversion to an image file can be performed by using an existing image conversion tool such as pdftoppm (https://www.xpdfreader.com/pdftoppm-man.html), and original conversion processing can be performed. It may be developed and incorporated.

Next, the coordinate information of the table caption extracted by the text data analysis unit in the process 300 (see Table 510 in FIG. 5B) is adjusted to the coordinate scale of the image file (process 612). Based on the size of the document page on the image file, the coordinate information of the table caption is converted from the coordinate information on the text file with the coordinate information to the coordinate information on the image file. The conversion can be performed by the following formula.

Abscissa of table caption (image file) =
Abscissa of table caption (text file) x (horizontal size of document page (image file) ÷ horizontal size of document page (text file))

Vertical coordinates of table caption (image file) =
Vertical coordinates of table caption (text file) x (vertical size of document page (image file) ÷ vertical size of document page (text file))

The size of the document page on the image file can be obtained by an image processing library such as OpenCV (https://opencv.org).

Next, the image analysis unit 203 performs a process of limiting the image range in which the table detection is executed (process 613). The image range is limited based on the coordinates of the table caption. In the flow of FIG. 6, only the page from which the caption is extracted is image-converted, but all the pages may be image-converted in advance.

FIG. 7 shows an example when the image analysis unit 203 limits the image range in which the table detection is executed. In image 700, assuming that the table caption is located at the upper left of the table, the range in which the table detection is executed is limited to the area 701. In image 710, the table caption is limited to the area 711 and the area 712 on the assumption that it is arranged in the upper left of the table.

As shown in image 710, when multiple table captions are included in the same page, the image range is such that the image range for which table detection is executed does not include multiple tables due to the positional relationship between the table captions. To limit.

The example assumes that the table caption is in the upper left corner of the table, but the position of the table caption is different for each document. When limiting the image range for executing table detection, it is necessary to change the process for limiting the image range in consideration of the positional relationship between the table caption and the table.

In this case, for example, the user may be configured to specify the positional relationship between the table caption and the image range in which the table detection is executed before the start of the process 600. Alternatively, processing 613 may be omitted and the entire page including the table caption may be used as the image range for executing table detection. However, using the positional relationship between the table caption and the image range in which the table detection is executed can be expected to reduce the processing amount and improve the accuracy.

Next, the image analysis unit 203 executes table detection from the image range limited by the process 613 (process 614). Any method may be used for table detection. For example, there is a method of performing rectangle detection using an image processing library or the like and detecting the detected rectangle as a table. In addition, straight line detection is performed using an image processing library, etc., and the uppermost straight line and the lowest straight line in the vertical coordinates are set as the upper and lower sides of the table, and the range surrounded by the upper and lower sides. May be detected as a table.

Next, the image analysis unit 203 executes layout recognition of the table detected in the process 614 (process 615). This process is executed for each table recognized in process 614. Any method may be used for table layout recognition.

FIG. 8A is a flow chart of an example of the table layout recognition process. For example, the process 615A of FIG. 8A shows an example of layout recognition of a table in which the range of each cell of the table is represented by ruled lines in the vertical and horizontal directions.

In this example, first, a rectangle detection process is executed for the range of the table detected in the process 614 by using an image processing library or the like (process 8110). Then, each detected rectangle is recognized as each cell in the table. Then, the coordinate information of each cell, for example, the abscissa and ordinate coordinates of the upper left point of the table, the width and height of the cell, is extracted using an image processing library or the like (process 8111). Then, based on this coordinate information, adjacent cells are determined for each cell in the vertical direction and the horizontal direction, and cells belonging to the same column and the same row are recognized (process 8112). The image analysis unit 203 executes the above processing by image analysis.

Process 615B of FIG. 8B shows an example of layout recognition of a table including only horizontal ruled lines.
FIG. 8C is a flow chart showing the details of the process 8212 of the process 615B.
In the example of process 615B, the image analysis unit 203 first performs histogram analysis on the range of the table detected in process 614 using an image processing library or the like (process 8210). A histogram is created by counting the number of pixels with colors in the vertical and horizontal directions of the table image. Then, each cell in the table is recognized by using the valley portion of the histogram as the cell boundary.

9A and 9B show an example of recognizing cells by histogram analysis in a table containing only horizontal ruled lines. Table 9110 of FIG. 9A shows an example of a table for which a histogram analysis is performed, and Table 9120 of FIG. 9B shows the result of a histogram analysis for Table 9110. In Table 9120, the area surrounded by the dotted line is recognized as a cell, and the coordinate information of each cell, for example, the abscissa and ordinate coordinates of the upper left point of the table, the width and height of the cell, the image processing library, etc. (Processing 8211).

Next, cells belonging to the same column and the same row are recognized (process 8212). FIG. 8C shows the details of the process 8212 for recognizing cells belonging to the same column and the same row in a table including only horizontal ruled lines. First, for each cell, the coordinate distance from the other cells is calculated (process 8310). Then, the cells having a closer distance in the vertical direction are set to the same column, and the cells having the closest distance in the horizontal direction are set to the same row (process 8311). However, this may lead to misrecognition of rows and columns.

Table 9210 of FIG. 9C shows the case where the columns of Table 9110 are correctly recognized. Columns 9213 and 9214 are connected to the merged cell 9217 and are recognized as the same column 9211. Similarly, column 9215 and column 9216 connect to merged cell 9218 and are recognized as identical column 9212.

Table 9220 of FIG. 9D shows the case where the columns of Table 9110 are erroneously recognized. Originally, column 9224 should be connected to merged cell 9227 and belong to column 9221 as well as column 9223. However, column 9224 is connected to column 9225 and column 9226 by the merged cell 9228 and is recognized as the same column 9222. This is because the cell having the closest coordinate distance to the cell 9229 has been determined to be the merged cell 9228.

In order to correct such misrecognition of columns and rows, in this embodiment, the data storage unit 204 performs a process of estimating the cell layout from the body of the PDF document. First, a process of extracting the text in each cell from a text file with coordinate information is performed (process 8312). Since the image analysis unit 203 extracts the text in each cell from the text file with the coordinate information, the coordinate information of each cell is changed from the coordinate information on the image file to the coordinate information on the text file with the coordinate information. Convert. The conversion is performed by the following formula.

Cell abscissa (text file) =
Abscissa of cell (image file) x (horizontal size of document page (text file) ÷ horizontal size of document page (image file))

Cell ordinate (text file) =
Cell vertical coordinates (image file) x (document page vertical size (text file) ÷ document page vertical size (image file))

Cell width (text file) =
Cell width (image file) x (horizontal size of document page (text file) ÷ horizontal size of document page (image file))

Cell height (text file) =
Cell height (image file) x (horizontal size of document page (text file) ÷ horizontal size of document page (image file))

The data storage unit 204 collates this coordinate information with the coordinate information in the text file with the coordinate information, and extracts the text existing within the range of the cell on the coordinates as the text in the cell. Taking file 400 as an example, the coordinate information of each text to which the TOKEN tag is attached is referred to, and the center coordinates of the text are obtained by the following formula.

Center coordinates (X, Y) = (Abscissa + (width / 2), Vertical coordinates + (height / 2))

If the center coordinates are within the coordinate range of the cell, it is determined that the text is the text in the cell.

Next, the text data analysis unit 202 extracts and refers to the explanation of the table from the text file with the coordinate information (process 8313), and corrects the rows and columns (process 8314). First, all the texts with the TOKEN tag are extracted from the text file with the coordinate information, and a sentence is formed based on the coordinate information of each text. This is done by recognizing continuous text on the coordinates as a series of sentences. It is checked whether the sentence formed in this way contains a sentence pattern explaining the structure of the table, and if it is included, one sentence containing the sentence pattern is determined to be the explanation sentence of the table. Extract.

FIG. 9E shows an example of a sentence pattern 9300 explaining the structure of the table. The sentence pattern 9300 is stored in advance in the disk device 104 or the like. In the example of the figure, there is one sentence pattern, but a plurality of sentence patterns may be prepared in advance, and if any of the sentence patterns is included, one sentence may be determined as the explanation of the table.

Next, the explanation of the table is analyzed. The explanatory text 9310 of FIG. 9E shows an example in which the explanatory text of Table 9110 is extracted by the sentence pattern 9300. In the sentence pattern 9300, a cell containing "XX" in the text in the cell and a cell containing "△△" in the text in the cell are connected by a merged cell containing "○○" in the text in the cell, and are in the same column. Is shown to form. Therefore, by referring to the explanation 9310, the merged cell containing "Step" in the text in the cell, the cell containing "Compound" in the text in the cell, and the cell containing "Yield" in the text in the cell are connected. You can see that.

Since the text corresponding to each cell is extracted in the process 8312, the cell connection relationship can be estimated based on the description 9310 by referring to the text in the cell of each extracted cell. For example, in the recognition of rows and columns based on the coordinate distance between cells by the image analysis unit, as shown in Table 9220 of FIG. 9D, misrecognition that the combined cell 9228 and the cell 9229 belong to the same column 9222 is erroneously recognized. Even if the above occurs, by collating the explanatory text 9310 with the text in the cell, it can be found that the cell connection relationship based on the explanatory text 9310 is inconsistent. Therefore, as shown in Table 9210 of FIG. 9C, the combined cell 9217 and the cell 9119 form the same column 9211, which can be corrected to the correct column recognition.

After the table layout recognition process is completed, the table coordinate information is output (process 616).

The coordinate information 1000 of FIG. 10 shows an output example of the coordinate information of the table whose layout is recognized by the table layout recognition process 515. A unique ID is assigned to each cell constituting the table, and the coordinate information of each cell (in the example of the coordinate information 1000, the abscissa and ordinate coordinates of the upper left part of the cell, the cell width, and the cell height) is output. This coordinate information is the coordinate information on the image file. Also, for each cell, the ID of the adjacent cell recognized by the table layout recognition process is output. Adjacent cells have the closest coordinate distance in the lateral direction in the cell group recognized as the same row and the closest coordinate distance in the lateral direction in the cell group recognized as the same column. Adjacent cells in the same row exist on the left and right, and adjacent cells in the same column exist on the top and bottom. When the cell is a merged cell, it may be adjacent to a plurality of cells on the left, right, top and bottom, respectively. Further, the coordinate information 1000 of the extracted table is given the corresponding caption of the table shown in the table 510. The coordinate information 1000 is sent from the image analysis unit 203 to the data storage unit 204.

When the processes 615 and 616 are executed for all the tables detected in the process 614, the table layout recognition process is terminated (process 617: Yes). If not, processing 615 and processing 616 are subsequently performed (process 617: No).

When the processes 610 to 617 are executed for all the pages for which the table caption is detected, the table detection / layout recognition process is terminated (process 618: Yes). If not, the processes 610 to 617 are subsequently performed (process 618: No).

FIG. 11 shows a process in which the data extraction program 200 according to the first embodiment extracts and outputs text in a table in a series of processes for extracting table data. This process is performed for the number of tables detected in process 614.

The data storage unit 204 extracts the text in the cell from the text file with the coordinate information for each cell based on the coordinate information 1000 of the table received from the image analysis unit 203 (process 1110). The method of extracting the text is the same as the method shown in the process 8312.

Next, the extracted text is shaped into data that reproduces the structure of the original table and output (process 1111). From the coordinate information of the table received from the image analysis unit 203, 1000, the adjacent cells of each cell are recognized, the structure of the original table is reproduced, and the data is output. The output format can be an HTML table representation or a DB table structure.

The HTML file 1210 of FIG. 12 is a representation of Table 1200 in HTML table representation. As shown in the HTML file 1210, a th tag or a td tag is added to each cell. Then, the cells in the same row are surrounded by tr tags and the whole is surrounded by table tags to express the structure of the table. When expressing the merged cells in the horizontal direction, the colspan attribute is added as shown in HTML file 1210, and when expressing the merged cells in the column direction, the rowspan attribute is given.

By adding a th or td tag to the text of each cell extracted in process 1110, referring to the adjacent cell of each cell recorded in the coordinate information 1000 of the table, and enclosing the cell in the same row with the tr tag. , HTML table representation can output the extracted table data. In the case of a merged cell with multiple adjacent cells in the column direction, the colspan attribute is added, and in the case of a merged cell with multiple adjacent cells in the row direction, the rowspan attribute is added.

The output format does not necessarily have to be an HTML table representation, and as already described, it may be a DB table structure or the like. At the time of output, the caption of the table is added as metadata. When outputting as an HTML table representation, a method such as adding a table caption as attribute information of the table tag can be considered. In the case of a DB table, it is possible to use the table caption as the table name. The output data is saved on the disk device 104 in the form of a file, DB, or the like.

FIG. 13 shows a process in which the data extraction program 200 according to the first embodiment extracts and outputs information indicating the properties of the table data in a series of processes for extracting the table data.

First, the text data analysis unit 202 extracts all the texts to which the TOKEN tag is attached in the text file with the coordinate information, and forms a sentence based on the coordinate information of each text. This is done by recognizing continuous text on the coordinates as a series of sentences. It is examined whether or not the sentence formed in this way contains a sentence pattern explaining the nature of the table (process 1310).

If it is included, one sentence containing the sentence pattern is extracted as information indicating the nature of the table and passed to the data storage unit. The data storage unit associates the passed information (text) with the table data output in FIG. 11 and saves it on the disk device 104 in the form of a file, DB, or the like (process 1311).

FIG. 14 is a table diagram showing a sentence pattern 1400, which is an example of a sentence pattern for extracting a text explaining the properties of the table.

By collating the table caption, which is the metadata of the table data output in process 1111, with the information extracted in process 1311, the application that executes data analysis at the time of data analysis can check the properties of the table data output in process 1111. , You can know.

In Example 1, an example of extracting a table in a digital document was shown. The object to be extracted is not limited to a table, but may be a graph, a drawing, or the like. In the second embodiment, an example of extracting a graph in a digital document will be described.

The outline of the computer system of the data extraction device according to the second embodiment will be described. The hardware configuration of the computer system according to the second embodiment may be the same as that in FIG. Since the second embodiment is an example of extracting graph data from a PDF file, the data extraction device 100 executes a data extraction program for graph extraction and extracts the graph data from the PDF file.

The functional block configuration of the data extraction program 200 stored in the memory 103 for graph extraction may be the same as in FIG. However, the specific processing executed by each functional block 2001-204 is partially different from that of the first embodiment. This will be described below with reference to FIG.

The conversion processing unit 201 reads the PDF file stored in the disk device 104 and converts it into a text file with coordinate information. In addition, the PDF file stored in the disk device 104 is read and converted into an image file. This process may be the same as in Example 1.

The text data analysis unit 202 analyzes the text file with the coordinate information and extracts the graph caption. In addition, a text file with coordinate information is analyzed to extract a text showing the properties of the graph.

The process of extracting the graph caption is the same as the flow shown in FIG. 3 of the first embodiment, but in the keyword search process 311, the caption of the graph such as "graph" or "Graph" is used for the text file with the coordinate information. Performs a process to search for text that contains keywords that are likely to be included in. Further, in the process 313 for collating with the sentence pattern, a sentence pattern for extracting a graph is used instead of the sentence pattern for extracting the table shown in FIG. 5A. As a specific example, it is a sentence pattern that explains the contents of "graph" and "Graph".

If the text around the extracted keyword is not a sentence of a sentence pattern, it is judged that the text including the keyword is a graph caption, and the same information as in FIG. 5B (however, the data related to the table is the data related to the graph. (Replace) is sent to the image analysis unit 203 (process 315). The series of processes from process 312 to process 315 is repeated by the number of texts obtained by the keyword search (process 316).

Further, the process of extracting the text showing the properties of the graph is the same as in FIG. 13, but instead of the sentence pattern for extracting the text showing the properties of the table shown in FIG. 14, the text showing the properties of the graph is used. Use a sentence pattern for extraction.

FIG. 15 is a table diagram showing a sentence pattern 2300, which is an example of a sentence pattern for extracting a text showing the properties of a graph.

The image analysis unit 203 analyzes the image file based on the graph caption information received from the text data analysis unit 202, and detects the graph.

The data storage unit 204 outputs the graph extracted by the image analysis unit 203 in the form of a file or the like.

FIG. 16 shows a process 2000 in which the data extraction program 200 detects and outputs a graph in a series of processes for extracting graph data. The graph detection / output process 2000 is executed for each page in which the graph caption is detected in the process 300.

First, the image analysis unit 203 refers to the table 510 created in the process 300 by the text data analysis unit 202 (however, the table caption is replaced with the graph caption), and confirms the page number of the page including the graph caption (process 2010). ). Then, the document page of the PDF file is converted into an image file (process 2011). The conversion to the image file is the same as in the first embodiment.

Next, a process is performed in which the coordinate information of the graph caption extracted by the text data analysis unit 202 in the process 300 is adjusted to the coordinate scale of the image file (process 2012). Based on the size of the document page on the image file, the coordinate information of the graph caption is converted from the coordinate information on the text file with the coordinate information to the coordinate information on the image file. The size conversion can be executed in the same manner as in the case of the table caption by replacing the "table caption" with the "graph caption" in the conversion formula of the first embodiment.

Next, the image analysis unit 203 performs a process of limiting the image range in which the graph detection is executed (process 2013). The image range is limited based on the coordinates of the graph caption. The method by which the image analysis unit 203 limits the image range in which the graph detection is executed is the same as in the first embodiment.

Next, the image analysis unit 203 executes graph detection from the image range limited by the process 2013 (process 2014). Any method may be used for graph detection. For example, there is a method of performing rectangle detection using an image processing library or the like and detecting the detected rectangle as a graph. Further, straight line detection may be performed using an image processing library or the like, two orthogonal straight lines may be recognized as the vertical axis and the horizontal axis of the graph, and the range surrounded by both straight lines may be detected as a graph. After detecting the graph, the image analysis unit 203 sends the image of the detected graph to the data storage unit 204.

The data storage unit 204 that has received the image of the graph stores it in the disk device 104 in the form of a file or the like. At the time of output, the caption of the graph is added as metadata. Any method can be used to add the metadata, but a method of using the caption of the graph as the file name can be considered.

Also in the second embodiment, similarly to FIG. 13 of the first embodiment, the data extraction program 200 may perform a process of extracting and outputting information indicating the properties of the graph data in a series of processes for extracting the graph data. it can.

First, the text data analysis unit 202 extracts all the texts to which the TOKEN tag is attached in the text file with the coordinate information, and forms a sentence based on the coordinate information of each text. This is done by recognizing continuous text on the coordinates as a series of sentences. It is checked whether the sentence formed in this way contains a sentence pattern (see FIG. 15) that explains the nature of the graph (process 1310), and if so, a sentence containing the sentence pattern is included. , As information indicating the nature of the graph, it is extracted and passed to the data storage unit 204. The data storage unit 204 stores the passed information (text) in the form of a file, DB, or the like on the disk device 104 (process 1311).

By collating the graph caption, which is the metadata of the graph data output in the process 2015, with the information extracted in the process 2211, the application that executes the data analysis at the time of data analysis can check the properties of the graph data output in the process 2015. , You can know.

According to the embodiment described in detail above, by using both the image information and the text with coordinate information, it is possible to accurately extract table data and graph data from, for example, a PDF file. In addition to the table data and the graph data, it is possible to extract information indicating the properties of the table and the graph. These improve the quality of data analysis.

The data extraction method described in the present specification can be applied not only to a PDF file but also to an object in a document page and a general digital document composed of coordinate information of the object. The techniques described in the examples improve the quality of data analysis for these digital documents.

Data extraction device 100, bus 101, CPU 102, memory 103, disk device 104, data extraction program 200, conversion processing unit 201, text data analysis unit 202, image analysis unit 203, data storage unit 204

Claims (15)

A conversion processing unit that converts an object on a document page and a digital document containing the coordinate information of the object into text data with image data and coordinate information.
A text data analysis unit that extracts captions related to the object to be extracted from the text data,
An image analysis unit that detects an object to be extracted based on the information of the caption extracted from the image data.
A data extraction system from digital documents. The text data analysis unit
Search for a text that is a candidate for a caption by collating the text data with a keyword, and identify whether the candidate for the caption is a caption or a part of the text by collating the candidate for the caption with a preset sentence pattern.
A data extraction system from the digital document according to claim 1. The conversion processing unit
Of the digital documents, the page from which the caption is extracted is converted into image data.
The image analysis unit
Perform object detection on the image data of the page from which the caption is extracted.
A data extraction system from the digital document according to claim 1. The conversion processing unit
Of the digital documents, the page from which the caption is extracted is converted into image data.
The image analysis unit
For the image data of the page from which the caption is extracted, the image range in which the image analysis is performed based on the coordinate information of the caption is limited, and the object is detected.
A data extraction system from the digital document according to claim 1. The object is table data or graph data,
A data extraction system from the digital document according to claim 1. The object you want to extract is a table
It has a data storage unit that collates the detected coordinate information of the table with the text data with the coordinate information and extracts the text in the table.
A data extraction system from the digital document according to claim 1. The image analysis unit
Extract each cell of the detected table and
The data storage unit
The coordinate information of each cell is collated with the text data with the coordinate information, and the text in each cell is extracted.
The data extraction system from the digital document according to claim 6. The text data analysis unit
By collating the text data with a preset sentence pattern, the text explaining the structure of the table is extracted.
The data storage unit
Recognize the layout of the table based on the text in the cell and the text describing the structure of the table.
A data extraction system from the digital document according to claim 7. The text data analysis unit
Extract information indicating the properties of an object from a text file with coordinate information based on a predefined sentence pattern.
A data extraction system from the digital document according to claim 1. The caption information is at least one of the page on which the caption was detected and the coordinates on which the caption was detected.
A data extraction system from the digital document according to claim 1. A data extraction method from a digital document that extracts a desired object from a digital document.
Text data generation processing, which generates text data with coordinate information from digital documents,
Caption detection processing, which detects the caption of the desired object from the text data by keyword search,
An object extraction process, which extracts the desired object from the image data of the page of the digital document in which the caption is detected.
A method of extracting data from a digital document. In the object extraction process
The analysis range for extracting the desired object in the image data is limited based on the coordinate information of the caption.
The method for extracting data from a digital document according to claim 11. Includes an image data conversion process that converts only the pages of the digital document in which the caption is detected into image data.
The method for extracting data from a digital document according to claim 11. When extracting a table as the object, the object extraction process
Cell detection processing that detects the position of a cell from the image data, and
Includes a text extraction process that extracts text corresponding to the cell position from the text data.
The method for extracting data from a digital document according to claim 11. The object extraction process
Layout estimation processing that estimates the layout of the table based on the detected position of the cell,
Table description reference processing that extracts and refers to the table description from the text data,
Includes a layout modification process that modifies the layout by collating the description of the table with the text corresponding to the cell position.
The method for extracting data from a digital document according to claim 14.

Images