CN117454851B - A method and device for extracting table data from PDF documents - Google Patents

Abstract

In the extraction method, after an initial form is obtained by analyzing a PDF document, a text list corresponding to a page where the initial form is located is firstly segmented to obtain a text two-dimensional list. Then, a table category of the initial table is determined based on the number of rows and columns of the initial table and the number of columns of the text two-dimensional list. Finally, the initial form is reconstructed based on the determined form category and the text list, and a reconstructed form is obtained as form data extracted from the PDF document. Therefore, the extraction efficiency and accuracy of the table data can be greatly improved.

Description

A method and device for extracting table data from PDF documents

Technical field

One or more embodiments of this specification relate to the field of computer technology, and in particular, to a method and device for extracting tabular data from PDF documents.

Background technique

In most cases, multi-source, heterogeneous and multi-dimensional supply chain data contain a wealth of valuable information, which is of great significance for guiding enterprise management, decision support and business model innovation. Among them, Portable Document Format (PDF), as a widely used unstructured data form, has significant advantages in cross-platform, high fidelity and security. Therefore, it is widely used in the production and production of various documents. It is widely used in communication. Especially in the field of enterprise applications, PDF documents are an important carrier for internal and external communication, such as prospectuses, regular reports of listed companies (including annual reports, semi-annual reports and quarterly reports, etc.), contract agreements, product instructions, etc. These PDF documents contain a large amount of corporate information, such as operating conditions, financial indicators, market competitiveness, product characteristics, etc., which are of great value to the company itself and its stakeholders. However, since PDF documents are usually not editable and contain a variety of unstructured data such as tables, images, and text, effectively extracting data from them becomes complex and time-consuming. Currently, methods for extracting data from PDF documents mainly include manual extraction and entry, PDF converters, open source tools, and intelligent algorithms. However, these methods have certain limitations and shortcomings. details as follows:

(1) Data complexity problem. PDF documents are usually composed of unstructured data with complexity and diversity such as tables, pictures, and texts. Common data conversion methods/tools are inefficient, costly, and fail to provide visual analysis functions, making operation inconvenient and usable. Restricted.

(2) Data quality issues. Due to various factors such as human subjective judgment, negligence or fatigue, manual extraction of unstructured data from PDF documents is prone to omissions, errors, and even the neglect of some important data information, which may have a negative impact on subsequent analysis applications.

(3) Data integrity issue. When using automated tools to extract data from PDF documents, often only some conventional financial indicator data can be extracted, while information such as financial notes, pictures, and texts that are extremely valuable for data analysis are ignored, affecting data integrity and analysis accuracy. sex.

(4) Data comparison problem. The structured data extracted from PDF documents manually is usually stored in Excel or Word document tables. When statistical analysis of year-on-year, month-on-month, and current year cumulative indicators is required in the future, historical data cannot be quickly retrieved and recalled.

(5) Data fusion issues. The structured data extracted from PDF documents using traditional data extraction methods/tools is usually difficult to reasonably summarize and store according to business themes, and the data usability is not strong, which brings challenges in data fusion.

In order to effectively solve the above problems, a more effective data extraction method for PDF documents needs to be provided.

Contents of the invention

One or more embodiments of this specification describe a method and device for extracting tabular data from PDF documents, which can greatly improve the efficiency and accuracy of extracting tabular data.

The first aspect provides a tabular data extraction method for PDF documents, including:

Parse the PDF document to obtain the initial table and multi-page text content contained in it;

Convert the multi-page text content into corresponding text lists, where a single text list includes multiple lines of text;

From each of the text lists, select the target text list corresponding to the page where the initial form is located;

Segment the target text list according to preset symbols to obtain a two-dimensional text list;

Determine the table category of the initial table based on the first row number and first column number of the initial table, and the second column number of the text two-dimensional list;

Determining the table type of the initial table includes determining that the table type is a three-line table if the first number of rows is less than a preset number of rows and the first column number and the second column number are equal. ; If the difference between the second column number and the first column number is equal to the preset column number, then the table category is determined to be a table with missing borders; if the difference between the second column number and the first column number is If the difference is greater than the preset number of columns, the table category is determined to be a color ladder table;

Reconstruct the initial table according to the determined table category to obtain a reconstructed table;

The reconstructed table is determined as table data extracted from the PDF document.

In the second aspect, a form data extraction device for PDF documents is provided, including:

The parsing unit is used to parse the PDF document and obtain the initial table and multi-page text content contained in it;

A conversion unit used to convert the multi-page text content into corresponding text lists, where a single text list includes multiple lines of text;

A selection unit configured to select the target text list corresponding to the page where the initial form is located from each of the text lists;

A segmentation unit, used to segment the target text list according to preset symbols to obtain a two-dimensional text list;

A determination unit configured to determine the table category of the initial table based on the first row number and first column number of the initial table and the second column number of the text two-dimensional list;

The determining unit is specifically configured to: if the number of first rows is less than the preset number of rows, and the number of first columns and the number of second columns are equal, determine that the table category is a three-line table; if the If the difference between the second column number and the first column number is equal to the preset column number, the table category is determined to be a table with missing borders; if the difference between the second column number and the first column number is greater than the preset Assuming the number of columns, the table category is determined to be a color ladder table;

A reconstruction unit, configured to reconstruct the initial table according to the determined table category to obtain a reconstructed table;

The determining unit is also used to determine the reconstructed table as table data extracted from the PDF document.

One or more embodiments of this specification provide a method and device for extracting table data from PDF documents. After parsing the initial table from the PDF document, the text list corresponding to the page where the initial table is located is first segmented to obtain A two-dimensional list of text. Then, based on the number of rows and columns of the initial table and the number of columns of the two-dimensional text list, the table category of the initial table is determined. Finally, based on the determined table category and the above text list, the initial table is reconstructed to obtain the reconstructed table as table data extracted from the PDF document. This can greatly improve the efficiency and accuracy of table data extraction.

Description of the drawings

In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of this specification. Those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of an implementation scenario disclosed in an embodiment of this specification;

Figure 2 shows a flow chart of a method for extracting table data from PDF documents according to one embodiment;

Figure 3 shows a schematic diagram of the PDF document parsing process in an example;

Figure 4 shows a text column representation diagram in one example;

Figure 5a shows a target text column representation diagram in an example;

Figure 5b shows a two-dimensional column representation of text in one example;

Figure 6 shows a schematic diagram of a table data extraction method for PDF documents in an example;

Figure 7a shows a schematic diagram of the document overview view in the visual analysis system;

Figure 7b shows a schematic diagram of the data extraction view in the visual analysis system;

Figure 7c shows a schematic diagram of the data conversion audit view in the visual analysis system;

Figure 8 shows a schematic diagram of a table data extraction device for PDF documents according to one embodiment.

Detailed ways

The solutions provided in this specification will be described below in conjunction with the accompanying drawings.

Generally speaking, the PDF documents of listed companies' periodic reports contain a wealth of data information, which is usually presented in tabular form, such as balance sheets, income statements, cash flow statements, and notes to financial reports. Extracting these tabular data from PDF documents can provide a more reliable data basis for corporate decision-making, make it easier to compare data at different time points or different companies, and better understand the financial changes of the benchmarking company. , thereby making corresponding plans and decisions more targeted.

In order to realize the automatic extraction of table data in PDF documents, existing solutions have proposed many document structured data conversion technologies.

PDF documents are usually stored in the form of images or binary encodings, and the document parsing method can be used to decode the document structure and parse the data type. Strouthopoulos et al. proposed a document parsing method based on PDF document structure and keywords, which can automatically identify and extract text information, and accurately determine paragraph boundaries and sentence integrity. Zhang et al. studied a rule-based document parsing method to convert PDF documents into XML format and extract metadata from them. Nguyen et al. introduced a method to convert PDF documents into image formats, using computer vision (CV) and image processing (IP) technologies to identify tables, pictures and text. Grijalva et al. developed a data conversion platform that first extracts text cells, bitmap images, and lines from scanned PDF documents, and then uses machine learning (machine learning) classification methods to parse the document content. Rizvi et al. proposed a BRExSys system framework using mask region-based convolutional neural network (Mask R-CNN) to analyze the page layout of PDF documents. In addition, Ahmed et al. also proposed a document parsing method based on multi-dimensional features such as text blocks, typography and geometric information. However, this method has low accuracy and requires more storage space and computing resources when processing large-scale PDF documents.

Data extraction refers to identifying and extracting specific types of information from tables, pictures, or text in PDF documents. For table data extraction, you first need to detect and understand the table structure, and then extract the data in it. Traditional methods mainly rely on predefined templates and rule matching to extract specific field content, but are limited by template creation and difficult to adapt to different table structures. The machine learning method uses image segmentation and recognition algorithms such as YOLO and UNet to detect the table structure, and then uses optical character recognition (optical character recognition, OCR) technology to extract the table data. Hashmi et al. proposed a method based on guided anchor points to accurately locate rows and columns in table images, with strong generalization capabilities. Jiang et al. proposed a deep learning model based on the table cell structure, which improves the accuracy of processing heterogeneous table data by learning the characteristics of different types and content cells of the table.

Different from the above methods, this solution proposes a process-based table data extraction method, which can not only extract table subject information in PDF documents, but also achieve structural analysis and data extraction of complex tables.

Figure 1 is a schematic diagram of an implementation scenario disclosed in an embodiment of this specification. In Figure 1, the PDF document is first parsed to obtain the initial table and text content. Afterwards, the initial table can be reconstructed based on the text content to obtain the extracted table data. Finally, the extracted tabular data can be visually displayed for data analysts to view and review, and the tabular data that has passed the review can be saved to the database.

Figure 2 shows a flow chart of a method for extracting table data from PDF documents according to one embodiment. The method can be executed by any device, device, platform, or device cluster with computing and processing capabilities. As shown in Figure 2, the method may include the following steps.

Step S202: The PDF document is parsed to obtain the initial table and text content of multiple pages contained therein.

In one embodiment, a Python-based open source tool (pdfplumber) can be used to parse the PDF document to obtain the initial table and the text content of multiple pages (referred to as multi-page text content). It should be understood that the text content of the multiple pages includes the text content of the page where the initial form is located.

Figure 3 shows a schematic diagram of the PDF document parsing process in an example. In Figure 3, for a given PDF document, it is first read in the form of a binary content stream and converted into a Python object, and then the PDF document is traversed page by page. Parse various objects such as lines, rectangles, points, images, and characters on the page. For tables, refer to the idea of the Nurminen algorithm. First, obtain the actual table lines based on information such as one-dimensional lines, two-dimensional rectangles, and connection points. Then, by analyzing the position information of text alignment, we can speculate on possible virtual lines, and then merge these. lines to build table cells; then further extract the text characters in the cells and save the table data as a two-dimensional list of text.

For text content, the text is first extracted based on methods such as PDFMiner, the text content stream is decoded to extract characters, and then the horizontal and vertical distances between characters are calculated, and spaces and line breaks are inserted between characters to reconstruct the text content structure. And save the text as a line-by-line string.

Returning to Figure 2, Figure 2 can also include the following steps:

Step S204: Convert multiple pages of text content into corresponding text lists. A single text list includes multiple lines of text.

In one embodiment, for any page of text content, the text content of the page can be cut into multiple lines of text according to line breaks, and then the multiple lines of text can be organized into a list to obtain a corresponding text list. In a more specific embodiment, the text list may also indicate the index (Index), data type (Type), size (Size), etc. of each line of text.

Figure 4 shows a text column representation diagram in one example. In Figure 4, the text list includes an Index column, a Type column, a Size column and a Value column. Among them, the content of the index column is a text identifier, which can be numbered starting from 0. The content of the type column is a text data type, such as a string (Str), etc. The content of the size column is the number of strings contained in the text. The contents of numeric columns are text (also called strings).

Step S206: Select the target text list corresponding to the page where the initial form is located from each text list.

As mentioned before, the text content of each page is converted into a corresponding text list. Here is the target text list converted by extracting the text content of the page where the initial table is located.

Step S208: Segment the target text list according to preset symbols to obtain a two-dimensional text list.

In one embodiment, the preset symbol here may be, for example, a blank.

As mentioned before, the target text list includes multiple lines of text, where each line of text is recorded in the form of a string. The above-mentioned segmentation of the target text list can be understood as dividing the string of each line into multiple substrings. , thus forming a sublist.

Figure 5a shows a representation of a target text column in an example. For the target text list in Figure 5a, after segmenting it, the resulting text two-dimensional list can be as shown in Figure 5b. In Figure 5b, the sublist corresponding to each row includes four substrings, and each substring is separated by a comma, so it can also be understood that the two-dimensional text list includes four columns.

It should be noted that due to the different formats of different PDF documents, there are often multiple table categories such as three-line tables, missing border tables, color ladder tables, cross-page tables, continuous tables, nested tables, and multi-head tables. For different table categories , the extraction methods are often different, so the table category is judged first below.

Step S210: Determine the table type of the initial table based on the number of rows and columns of the initial table and the number of columns of the two-dimensional text list.

Specifically, if the number of rows of the initial table D _t is less than the preset number of rows (for example, 2), and the number of columns of the initial table D _t is equal to the number of columns of the text two-dimensional list D _l,t , then the initial table D _t is determined The table type of is a three-line table; if the difference n between the number of columns of the text two-dimensional list D _l,t and the number of columns of the initial table D _t is equal to the preset number of columns (for example, 2), then determine the table of the initial table D _t The category is a table with missing borders; if the difference n between the number of columns of the text two-dimensional list D _l,t and the number of columns of the initial table D _t is greater than the preset number of columns, then the table category of the initial table D _t is determined to be a color ladder table.

It should be noted that since the above initial table is parsed using an open source tool, the table parsed using this open source tool may have the following problems: Three-line tables usually use three horizontal lines to distinguish the table header and table body, but this method may change the table. The body part is recognized as one row; the border missing table (also called the missing table at both ends) usually lacks the lines on the left and right sides of the table, but this method can only identify the middle part of the table; the color ladder table usually uses different shades of colors to distinguish adjacent rows , but this method is not sensitive to table color, and it is easy to identify two adjacent rows of data as the same cell.

In view of the above initial tables parsed through open source tools, there are corresponding problems. To this end, this plan will reconstruct the initial form.

Step S212: Reconstruct the initial table according to the determined table category to obtain the reconstructed table.

Specifically, for the three-line table, each row of the corresponding area of the initial table in the target text list is segmented according to spaces, and the target number of columns is determined by clustering several one-dimensional lists obtained by segmentation. , and correspondingly fill in the contents of the initial table into a table with the target number of columns and the number of rows contained in the corresponding area to obtain the reconstructed table.

The method for determining the corresponding area of the initial table in the target text list may include matching the first i rows (for example, the first 2 rows) of the initial table with each row in the target text list (for example, calculating similarity), To determine the starting row of the initial table in the target text list. After that, it is judged line by line from the starting line downwards whether there are lines with spaces in the target text list. If there are no lines in a line, the line is used as the termination line of the initial table in the target text list. Finally, based on the determined starting line and ending line, the corresponding area of the initial table in the target text list can be determined.

In addition, several one-dimensional lists obtained by the above segmentation can also be regarded as several cells. By using clustering algorithms based on geographical location such as Kmeans to cluster the cells segmented for each row, we can obtain The number of target columns above. It should be understood that a new table can be obtained based on the target number of columns and the number of rows contained in the above corresponding area.

Finally, the above-mentioned corresponding filling of the contents in the initial table into a table with the target number of columns and the number of rows contained in the corresponding area specifically includes filling in the contents of each cell in the initial table into the new table. corresponding position in . For example, fill the i-th row and j-th column of the new table with the contents of the i-th row and j-th column of the initial table. It should be understood that after filling in the contents of each cell in the initial table into the new table, the reconstructed table corresponding to the initial table can be obtained.

Of course, in practical applications, after filling in the contents of each cell in the initial table into the new table, you can also determine whether there is a difference in the row spacing of the new table. Focus on the row spacing difference (position difference) and the first cell Check whether the positions of the cells are aligned, etc. to determine whether there are rows across the same row, and perform row merging for such situations. Finally, the newly created table after row merging is determined as the reconstructed table.

For the color ladder table, the reconstruction method is similar to the three-line table. The difference is that before dividing the initial table in the corresponding area of the target text list, the initial table can be preprocessed, for example, removing the None column, the None column here means that the corresponding column only includes None (null value), or includes both None and null.

For tables with missing borders, you can fill in the left and right columns of the initial table, and use None to fill in the missing content in the initial table after filling the columns to obtain the corresponding reconstructed table.

Step S214, determine the reconstructed table as table data extracted from the PDF document.

It should be noted that this solution can obtain accurate table data by reconstructing the initial table extracted from the PDF document.

Of course, in practical applications, in addition to obtaining the table data itself, it is also necessary to obtain the table name, unit of measurement, currency unit and other topic information associated with the table. The method for obtaining this topic information is explained below.

Match the first i rows of the initial table D _t with the target text list L _p,i to determine the starting row P _s of the initial table D _t in the target text list L _p,i . Determine whether the number m of all lines starting from the starting line P _s in the target text list L _p,i is not less than the preset number ρ. If it is not less than the preset number ρ, based on the starting line P _s and the preset number ρ, and extract the corresponding area from the target text list L _p,i as the area where the table topic information is located. Specifically, the above-mentioned corresponding area refers to the preset number ρ lines starting from the starting line P _s in the target text list L _p,i . If it is less than the preset number ρ, calculate the difference ρ-m between the preset number ρ and m, and use the difference ρ-m, the target text list L _p,i and other text lists L _p,i-1 , determine the area where the table subject information is located. Among them, other text lists L _p,i-1 are text lists corresponding to the text content of the previous page of the page where the initial table is located. The table topic information of the initial table D _t is determined by extracting keywords from the area where the table topic information is located.

Among them, based on the difference value ρ-m, the target text list L _p,i and other text lists L _p,i-1 , the specific area where the table subject information is located is determined to be from the end of the other text list L _p,i-1. A line starts ρ-m lines forward, as the preceding supplementary content of the target text list L _p,i . The target text list L _p,i after adding supplementary content is determined as the area where the table topic information is located.

At this point, for the PDF document, the table data and table subject information of each page have been extracted.

Since tables in PDF documents may be displayed across pages, it is necessary to determine whether the extracted table data of two adjacent pages or more is a cross-page table (referred to as a cross-page table) or a continuous table (referred to as a continuous table). ), and use corresponding methods to restore and merge them. Among them, the continuous tables are merged because the tables on the other pages of the continuous table except the first page table have no subject information, so they need to be merged to ensure the completeness and accuracy of the subject information of the tables, so as to better process the data. Fusion and comparative analysis.

The following describes the judgment and merging process of the above-mentioned cross-page and continuous tables.

Assume that through the method shown in Figure 2, the extracted table data includes a first reconstructed table D _t,-1 and a second reconstructed table D _t,1 , and the first reconstructed table D _t,-1 is located in the previous page, the second reconstructed table D _t,1 is located on the next page, then it can first be judged whether the first condition is met. The first condition here may include that the last row of the first reconstructed table D _t,-1 matches the last row of the corresponding first text list L _t,i-1, and that the last row of the second reconstructed table D _t,1 The first row matches the corresponding first row of the second text list L _t,i , and the number of columns of the first reconstructed table D _t,-1 is equal to the number of columns of the second reconstructed table D _t,1 ( Or the header data of the first reconstructed table D _t,-1 is consistent with that of the second reconstructed table D _t,1 ). That is to say, the first condition includes three constraints.

If the first condition is met, it is determined that the first reconstructed table and the second reconstructed table are cross-page tables. If the first condition is not met, it is determined that the first reconstructed table and the second reconstructed table are two independent tables. sheet.

When the first reconstructed table and the second reconstructed table are cross-page tables, it can be determined whether the similarity between the last row of the first reconstructed table and the first row of the second reconstructed table is greater than a preset threshold. σ, if so, it is determined that the first reconstructed table and the second reconstructed table are cross-row cross-page tables, so that after removing the duplicate header data (that is, removing the header data of the second reconstructed table), the first reconstructed table can be merged. A reconstructed table and the second reconstructed table are combined to obtain a merged table. And if the above similarity is not greater than the preset threshold σ, it means that the first reconstructed table and the second reconstructed table are peer cross-page tables, so the last row and the second row can be intercepted from the first and second reconstructed tables respectively. One row is merged, and then the remaining parts of the first and second reconstructed tables are merged to obtain the merged table.

In addition, the continuous table is also a special cross-page table, in which the sub-table occupies a full page, that is, the contents of the initial table and the text list are consistent. The processing method is similar to that of the cross-page table, and will not be described again here.

It should also be noted that the merged table obtained through the above method may be a complex structure table, such as a nested table or a multi-head table, etc. For complex structure tables, this solution can also split them.

Specifically, for the above merged table, it can be determined whether there is an intermediate row that contains only one non-None row in the merged table. If yes, it is determined that the merged table is a nested table, so that the merged table can be split into upper and lower parts based on the intermediate behavior; if not, no splitting process is performed.

In this solution, after the merged table is split, for each split table, it can be further determined whether it is a multi-head table, which is explained below.

Assuming that the upper and lower parts obtained after splitting the merged table include: the first split table and the second split table, you can obtain the header data for the first split table (or the second split table), and Determine whether the number of rows of the header data is greater than 1 row. If it is greater, and one of the rows contains None and the other row does not contain None, then it is determined that the first split table (or the second split table) is a multi-head table, so Merge the two rows to get the target table.

It should be noted that the reason why the above operation of merging rows is performed when the first/second split table is judged to be a multi-head table is because the strategy adopted in the process of parsing the document using open source tools is with the finest granularity. lines to predict cells, causing merged cells in the long table to be split and filled with None.

It should also be noted that the reconstructed table, merged table, split table or target table obtained through the embodiments of this specification can be saved in the format of a CSV file, or can be further converted into a JSON format and stored in a database. Subsequently, according to actual needs, data analysis tools can be used to perform operations such as data cleaning, statistical analysis, and visual display, so as to gain a deeper understanding of the financial status and business conditions of the benchmarking enterprise.

Figure 6 shows a schematic diagram of a table data extraction method for PDF documents in an example. In Figure 6, after obtaining the initial table and its corresponding text list, the table subject information can be extracted from the initial table, including the table name, measurement unit, currency unit, etc. In addition, for the initial table, data extraction of irregular tables (such as three-line tables, tables with missing borders, color ladder tables, etc.), cross-page tables and continuous tables, as well as complex structure tables (such as nested tables and multi-head tables, etc.) can be carried out in sequence. ) data extraction, etc. The finally extracted table data includes table subject information and the table data itself, where the table data itself can be stored in JSON format or other forms.

In this solution, the extracted tabular data can also be displayed to users, and users can review and analyze the extracted tabular data.

In one embodiment, the extracted tabular data can be displayed through a visual analysis system. The visual analysis system can include three views: document overview view, data extraction view and data transformation review view. Among them, the document overview view is used to display PDF documents. The data extraction view is used to display the distribution of tables of different table categories extracted from PDF documents. The data conversion review view is used to review tabular data extracted from PDF documents.

The above three views are described in detail below.

Figure 7a shows a schematic diagram of the document overview view in the visual analysis system. The document overview view includes an a1 area and a2 area. The a1 area is used to display the PDF document, and the a2 area uses a two-layer tree structure to display each chapter in the PDF document. Document elements such as tables, pictures, and text in sections and sections form an overview. The root node represents the document name, the leaf nodes represent each chapter of the document, and the chapter name is displayed on the connection line between the root node and the leaf node. The size of the leaf node indicates the number of document elements in the corresponding chapter. The leaf nodes use a circular treemap to display the sections contained in the chapter. Each pie chart represents a section. The size of the pie chart represents the number of document elements corresponding to the section. The pie chart encodes the tables and pictures of the corresponding section. and the amount of text. When the mouse is hovered over a pie chart, the name of the corresponding section will be displayed. Click on a chapter name or a pie chart to jump to the corresponding location in the PDF document.

Figure 7b shows a schematic diagram of the data extraction view in the visual analysis system. The left side of the view shows different tables such as standard tables, three-line tables, missing border tables, color ladder tables, cross-page tables, continuous tables, nested tables, and multi-head tables. Illustration of the type of form. The right side shows the total number and review status of the corresponding type of form in the form of a bar chart. Users can select the table type they want to view on the right and click on the corresponding bar of the histogram of interest to further view the review status.

Figure 7c shows a schematic diagram of the data conversion review view in the visual analysis system. This view supports users to review and analyze the extracted table data. Users can interactively select filters in the data extraction view to view, trace, analyze and correct the extracted table data. For the review of tabular data, users can sort by clicking on the column header of each column, and drag the column header to move left and right to change the order of the columns, so as to organize the table content according to personal analysis habits. The "magnifying glass" in the upper right corner of the data table indicates conversion traceability. Click the "magnifying glass" icon, and the original PDF document corresponding to the data table will be highlighted in the document overview view, making it easier for users to analyze the data before and after conversion, and review and confirm it. Accuracy of data.

Specifically, the above-mentioned data conversion audit view may include four areas c1-c4, and these four areas will be described below.

As shown in area c1, for the extracted table data, when the user hovers the mouse over a certain data row, the "Edit" and "Remarks" icons will be displayed on the right side. The user can click the "Edit" icon to modify and Record, or click the "Remarks" icon to record the content directly for accuracy. As shown in area c2, data rows that are accurate for review will be marked with a light gray background. The "Remarks" icon will be displayed on the right side of the audited data row, and you can click to view the audit log at any time. In addition, as shown in area c3, if the user finds an error in the data and clicks the "Edit" icon on the right side of the data row, the data row will be marked with a dark gray background, and a modified row with a light gray background will be inserted at the bottom. And copy the error row data as it is, and each data cell can be edited. The user can modify it directly, and the modified data will be displayed in bold. Finally, as shown in area c4, by clicking the "Remarks" icon on the right side of the data row, the user can record the modification log, including information such as whether the data is correct and audit notes.

Based on the above, this solution first parses the obtained PDF document, extracts the tables, and then reconstructs the extracted tables to achieve data conversion. Specifically, for tabular data, this solution uses data extraction methods to obtain the subject information of the form and the tabular data itself. In order to further improve the quality of data conversion and address possible data accuracy and efficiency issues during the data conversion process, this solution also provides a visual analysis system that enables data comparison, traceability, and analysis. Finally, the converted structured data is integrated into the database to facilitate future retrieval and retrieval.

All in all, this plan designed a set of intelligent processing strategies for PDF documents with special content structure and style characteristics for the periodic reports of listed companies, which improved the quality and efficiency of structured conversion processing of PDF documents. A new visual analysis system was built to display the extracted tabular data. In addition, the visual analysis system also supports users to review and analyze the extracted tabular data.

Corresponding to the above-mentioned table data extraction method for PDF documents, one embodiment of this specification also provides a table data extraction device for PDF documents. As shown in Figure 8, the device may include:

The parsing unit 802 is used to parse the PDF document to obtain the initial table and multi-page text content contained therein.

The conversion unit 804 is used to convert multiple pages of text content into corresponding text lists, where a single text list includes multiple lines of text.

The selection unit 806 is used to select the target text list corresponding to the page where the initial table is located from each text list.

The segmentation unit 808 is used to segment the target text list according to preset symbols to obtain a two-dimensional text list.

The determining unit 810 is configured to determine the table type of the initial table based on the first row number and first column number of the initial table, and the second column number of the text two-dimensional list.

The determination unit 810 is specifically configured to: if the number of the first rows is less than the preset number of rows, and the number of the first columns is equal to the number of the second columns, determine the table type to be a three-line table; if the number of the second columns is equal to the number of the first columns, If the difference in the number of columns is equal to the preset number of columns, the table category is determined to be a table with missing borders; if the difference between the number of the second column and the number of the first column is greater than the preset number of columns, the table category is determined to be a color ladder table.

The reconstruction unit 812 is used to reconstruct the initial table according to the determined table category to obtain a reconstructed table.

The determining unit 810 is also used to determine the reconstructed table as table data extracted from the PDF document.

In one embodiment, the number of reconstructed tables is two, and the two reconstructed tables include a first reconstructed table located on the previous page and a second reconstructed table located on the next page; the device further includes :

The judgment unit 814 is used to judge whether the first condition is met. The first condition includes that the last row of the first reconstructed table matches the last row of the corresponding first text list, and the first row of the second reconstructed table matches The first row of the corresponding second text list matches, the number of columns of the first reconstructed table and the second reconstructed table are equal, or the header data of the first reconstructed table and the second reconstructed table are consistent;

The judgment unit 814 is also used to judge whether the similarity between the last row of the first reconstructed table and the first row of the second reconstructed table is greater than the preset threshold when the above-mentioned first condition is met. If so, then After removing duplicate header data, merge the first reconstructed table and the second reconstructed table to obtain a merged table; if not, merge the last row of the first reconstructed table and the first row of the second reconstructed table, Get the merged form.

In one embodiment, the device further includes: a splitting unit 816;

The judgment unit 814 is also used to judge whether there is an intermediate row in the merged table that contains only one non-None row;

The splitting unit 816 is used to split the merged table into upper and lower parts based on the middle row that contains only one non-None row in the merged table.

In one embodiment, the above two parts include a first split table and a second split table; the device further includes: a merging unit 818;

The merging unit 818 is used to obtain the header data of the first/second split table. If the number of rows of the header data is greater than 1 row, and one row contains None and the other row does not contain None, then merge With these two lines, the target table is obtained.

In one embodiment, the reconstruction unit 812 is specifically used to:

When the table category is a three-line table or a color ladder table, segment each row of the corresponding area of the initial table in the target text list according to spaces, and aggregate several one-dimensional lists obtained by segmentation. class, determine the target number of columns, and fill in the contents of the initial table into a table with the target number of columns and the number of rows contained in the corresponding area above to obtain the reconstructed table;

When the table category is a table with missing borders, fill in the left and right columns of the initial table, and fill the missing content in the initial table after filling the columns with None to obtain the corresponding reconstructed table.

In one embodiment, the device further includes:

Matching unit 820, used to match the first i rows of the initial table with the target text list to determine the starting row of the initial table in the target text list;

Extraction unit 822, configured to extract the corresponding region from the target text list according to the starting line and the preset number when the number of all lines starting from the starting line in the target text list is not less than a preset number. As the area where the table subject information is located;

The determination unit 810 is also used to calculate the difference between the preset number and the total number of rows when the total number of rows is less than the preset number, and determine the location of the table subject information based on the difference, the target text list and other text lists. area; wherein, the other text list is a text list corresponding to the text content of the previous page of the page where the initial table is located;

The determining unit 810 is also used to determine the table theme information by extracting keywords from the area where the table theme information is located.

In one embodiment, the determining unit 810 is specifically used to:

The difference lines starting from the last line of other text lists will be used as the previous supplementary content of the target text list;

The target text list after adding supplementary content is determined as the area where the table topic information is located.

In one embodiment, the conversion unit 804 is specifically used to:

For a certain page of text content, the text content of the page is cut into multiple lines of text according to line breaks, and the multiple lines of text form a corresponding text list.

The functions of each functional unit of the device in the above embodiments of this specification can be realized through each step of the above method embodiment. Therefore, the specific working process of the device provided in one embodiment of this specification will not be described again here.

An embodiment of this specification provides a table data extraction device for PDF documents, which can greatly improve the efficiency and accuracy of table data extraction.

Each embodiment in this specification is described in a progressive manner. The same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on its differences from other embodiments. In particular, for the server embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment.

The steps of the method or algorithm described in conjunction with the disclosure of this specification may be implemented in hardware, or may be implemented in a processor executing software instructions. Software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, mobile hard disks, CD-ROM or any other form of storage well known in the art. in the medium. An exemplary storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage media may be located in an ASIC. Alternatively, the ASIC can be located in the server. Of course, the processor and storage media can also exist as discrete components in the server.

Those skilled in the art should realize that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware, or any combination thereof. When implemented using software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Storage media can be any available media that can be accessed by a general purpose or special purpose computer.

The foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desired results. Additionally, the processes depicted in the figures do not necessarily require the specific order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain implementations.

The above-mentioned specific implementations further describe the purpose, technical solutions and beneficial effects of this specification. It should be understood that the above are only specific implementations of this specification and are not intended to limit the scope of this specification. The scope of protection: any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions in this manual shall be included in the scope of protection of this manual.

Claims (9)

1. A table data extraction method for PDF documents, including: Parse the PDF document to obtain the initial table and multi-page text content contained in it; Convert the multi-page text content into corresponding text lists, where a single text list includes multiple lines of text; From each of the text lists, select the target text list corresponding to the page where the initial form is located; Segment the target text list according to preset symbols to obtain a two-dimensional text list; Determine the table category of the initial table based on the first row number and first column number of the initial table, and the second column number of the text two-dimensional list; Determining the table type of the initial table includes determining that the table type is a three-line table if the first number of rows is less than a preset number of rows and the first column number and the second column number are equal. ; If the difference between the second column number and the first column number is equal to the preset column number, then the table category is determined to be a table with missing borders; if the difference between the second column number and the first column number is If the difference is greater than the preset number of columns, the table category is determined to be a color ladder table; Reconstruct the initial table according to the determined table category to obtain a reconstructed table; Determine the reconstructed table as table data extracted from the PDF document; The reconstruction of the initial form includes: When the table type is a three-line table or a color ladder table, for each row of the corresponding area of the initial table in the target text list, segment it according to spaces, and obtain the Cluster several one-dimensional lists to determine the target number of columns, and correspondingly fill in the content in the initial table into a table with the target column number and the number of rows in the corresponding area to obtain the reconstruction sheet; When the table type is a table with missing borders, fill in the left and right columns of the initial table, and fill the missing content in the initial table after filling the columns with None to obtain the corresponding reconstructed table. 2. The method according to claim 1, wherein the number of the reconstructed tables is two, and the two reconstructed tables include a first reconstructed table located on the previous page and a third reconstructed table located on the next page. 2. Reconstruct the table; the method also includes: Determine whether the first condition is met. The first condition includes: the last row of the first reconstructed table matches the last row of the corresponding first text list; the first row of the second reconstructed table matches the corresponding The first row of the second text list matches; the number of columns of the first reconstructed table and the second reconstructed table are equal, or the number of columns of the first reconstructed table and the second reconstructed table The header data is consistent; When the first condition is met, determine whether the similarity between the last row of the first reconstructed table and the first row of the second reconstructed table is greater than a preset threshold, and if so, remove the After repeating the header data, merge the first reconstructed table and the second reconstructed table to obtain a merged table; if not, obtain a merged table by merging the last row and the first row. 3. The method of claim 2, further comprising: Determine whether there is an intermediate row in the merged table that contains only one non-None row; If so, the merged table is divided into upper and lower parts based on the middle line. 4. The method of claim 3, wherein the two parts comprise a first split table and a second split table; For the first/second split table, get the header data. If the number of rows of the header data is greater than 1 row, and one row contains None and the other row does not contain None, merge the two rows to get the target sheet. 5. The method of claim 1, further comprising: Match the first i rows of the initial table with the target text list to determine the starting row of the initial table in the target text list; When the number of all lines in the target text list starting from the starting line is not less than a preset number, extract the corresponding area from the target text list based on the starting line and the preset number. As the area where the table subject information is located; When the total number of lines is less than the preset number, the difference between the preset number and the total line number is calculated, and based on the difference, the target text list and other text lists, determine the The area where the table subject information is located; wherein the other text lists are text lists corresponding to the text content on the previous page of the page where the initial form is located; The table theme information is determined by extracting keywords from the area where the table theme information is located. 6. The method according to claim 5, wherein determining the area where the table subject information is located includes: The difference lines starting from the last line of the other text list and going forward are used as the previous supplementary content of the target text list; The target text list after adding supplementary content is determined as the area where the subject information of the table is located. 7. The method of claim 1, wherein converting the multiple pages of text content into multiple text lists includes: For a certain page of text content, the text content of the page is cut into multiple lines of text according to line breaks, and the multiple lines of text form a corresponding text list. 8. A visual analysis system, including: Document overview view, used to display the target PDF document; A data extraction view, used to display the distribution of different table categories extracted from the target PDF document; A data conversion review view is used to display tabular data extracted from the target PDF document according to the method of claim 1. 9. A form data extraction device for PDF documents, including: The parsing unit is used to parse the PDF document and obtain the initial table and multi-page text content contained in it; A conversion unit used to convert the multi-page text content into corresponding text lists, where a single text list includes multiple lines of text; A selection unit configured to select the target text list corresponding to the page where the initial form is located from each of the text lists; A segmentation unit, used to segment the target text list according to preset symbols to obtain a two-dimensional text list; A determination unit configured to determine the table category of the initial table based on the first row number and first column number of the initial table and the second column number of the text two-dimensional list; The determining unit is specifically configured to: if the number of first rows is less than the preset number of rows, and the number of first columns and the number of second columns are equal, determine that the table category is a three-line table; if the If the difference between the second column number and the first column number is equal to the preset column number, the table category is determined to be a table with missing borders; if the difference between the second column number and the first column number is greater than the preset Assuming the number of columns, the table category is determined to be a color ladder table; A reconstruction unit, configured to reconstruct the initial table according to the determined table category to obtain a reconstructed table; The determining unit is also used to determine the reconstructed table as table data extracted from the PDF document; The reconstruction unit is specifically used for: When the table type is a three-line table or a color ladder table, for each row of the corresponding area of the initial table in the target text list, segment it according to spaces, and obtain the Cluster several one-dimensional lists to determine the target number of columns, and correspondingly fill in the content in the initial table into a table with the target column number and the number of rows in the corresponding area to obtain the reconstruction sheet; When the table type is a table with missing borders, fill in the left and right columns of the initial table, and fill the missing content in the initial table after filling the columns with None to obtain the corresponding reconstructed table.