TWI414950B - System and method for parsing texts - Google Patents

Abstract

A method for parsing texts is disclosed. The method includes steps of: creating a text-described file; loading the text-described file and the text to be parsed; parsing the text according to the text-described file and extracting data from the text; outputting the data. A related system is also disclosed.

Description

Text analysis system and method

The present invention relates to a text processing system and method, and more particularly to a text parsing system and method.

The information system usually transmits information through text, and the information transmitted through the text is platform-independent, portable, and compatible, and the readability is also in line with the human visual understanding. The purpose of text analysis is to decompose the text, extract the data from it, to facilitate secondary processing, or to perform deep processing and services such as retrieval and text mining. As shown in the electronic report shown in Figure 1, the electronic report contains multiple forms (only Form 1 and Form 2 are listed in Figure 1), each form is a separate part, with title, source, date of tabulation, vendor number, The name of the manufacturer and the data sheet are composed. The data sheet contains multiple records. Each record includes the receipt number, the date of receipt, the item number, the purchase order number, the quantity received, the unit price of the goods received, and the electronic report such as this is analyzed and counted. , must be the information in the report (such as the title, source, date of the table, manufacturer number, manufacturer name and receipt number of each record, receipt date, item number, purchase order number, receipt quantity The receipt unit price is extracted and reorganized into the data structure that the information system can recognize according to its meaning. The conventional text decomposition process is difficult to standardize and must be customized by the programmer, which is prone to errors, especially when the text format is complex and the amount of data is large.

In view of the above, it is necessary to propose a general text analysis system and method.

A text parsing system, the system comprising: a loading module, configured to load a text description definition document and a text to be parsed, the text description defining respective blocks of the document definition text and matching rules of each material, each The block forms a multi-layered tree structure, the top layer is the root block, and the bottom layer is the smallest unit block; the parsing module is used to start from the root block and match each block of the text according to the matching rule of each block, if the matching The out-out block includes the sub-block, and the sub-block is matched in the matched block according to the matching rule of the sub-block until all the blocks are matched, and the minimum is obtained according to the matching rule of the corresponding data. Each piece of data is extracted from the unit block; and an output module is used to output the extracted data in a document format required by the user.

A text parsing method, the method comprising the steps of: creating a text description definition document, wherein the text description defines a matching rule for defining each block and each material of the document, each block forming a multi-layered tree structure, and the top layer is a root block, The bottom layer is the smallest unit block; the loaded text description defines the document and the text to be parsed; starting from the root block, each block of the text is matched according to the matching rule of each block, and if the matched block contains the sub-block And matching the sub-blocks in the matched blocks according to the matching rule of the sub-blocks until all the blocks are matched, and extracting each data from each of the smallest unit blocks according to the matching rule of the corresponding data ; and output the extracted data in the document format required by the user.

Compared with the prior art, the present invention combines the extensible markup language and the regular expression to describe the structure and matching rules of the text, decomposes the text according to the described text structure and matching rules, and extracts data from the text, thereby realizing various kinds of Solution to type text Analysis.

11‧‧‧Loading module

12‧‧‧analysis module

13‧‧‧Output module

S301‧‧‧Create text description definition document

S302‧‧‧Load text description definition document

S303‧‧‧Load text

S304‧‧‧Extracting information

S305‧‧‧Output data

Figure 1 is a schematic diagram of an electronic report.

2 is a functional block diagram of a preferred embodiment of the text analysis system of the present invention.

3 is a flow chart of a preferred embodiment of the text parsing method of the present invention.

Referring to FIG. 2, it is a functional block diagram of a preferred embodiment of the text analysis system of the present invention. The system includes a loading module 11, an analysis module 12, and an output module 13. The text parsing system runs on a computer, the system decomposes the text according to the text description definition document, extracts the data from the text, and outputs the extracted data. The data is specific information that needs to be extracted from the text. Taking the electronic report in Figure 1 as an example, the title of each form (such as the purchase order of *** Co., Ltd.), the source (such as H5S00001), and the tabulation are required. Date (such as 20070601 17:32:03), manufacturer number (such as 9876543210), manufacturer name (such as *** electronic company), and the receipt number of each record (such as HaA-012345), the date of receipt (such as 20070412) ), item number (such as 987654J00-001-BB), purchase order number (such as Ord-111111), receipt quantity (such as 2,400.00), receipt unit price (such as 12.45000) and other data extracted.

The text description defines the structure of the document description text and the matching rules. In this embodiment, the text description definition document is an Extensible Markup Language (XML) document, ie, *.xml. The text description defines a block in which the document defines a plurality of hierarchical tree structures to describe the structure of the text. The top layer block is the root block, and the root block contains several blocks, each of which contains several blocks, thereby finally forming a multi-layered tree structure. For any non-root block, there is only one zone The block directly contains the non-root block (called a sub-block). If a block contains multiple sub-blocks of the same structure, the sub-block family table type is defined. The bottom block does not contain sub-blocks, the bottom block is called the smallest unit block, and the smallest unit block contains only data. For example, for the electronic report in FIG. 1, the root block is represented by root, corresponding to the entire electronic report, and the sub-block of the root block is represented by a table, and the table is defined as a list type, corresponding to each form in the electronic report, and the table includes The sub-block title, from, date, supplierId, supplierName, and form respectively indicate the title, source, tabulation date, vendor number, vendor name, and data table of each form; form includes item sub-block, and item is defined as list type. Corresponding to each record in the data table; item includes sub-blocks consignId, consignDate, productId, PoId, inAmount, and price, respectively indicating the receipt number, receipt date, item number, purchase order number of each record, Receipt quantity, receipt unit price. The title, from, date, supplierId, supplierName, consignId, consignDate, productId, PoId, inAmount, and price are the smallest unit blocks, and the blocks contain data. For example, in Form 1 of Figure 1, title is "*** Ltd. Purchase Order", from "H5S00001", date is "20070601 17:32:03", supplierId is "9876543210", supplierName is "*** Electronic "Company", the first record in Form 1 has a consignId of "HaA-012345", a consignDate of "20070512", a productId of "987654J00-001-BB", a PoId of "Ord-111111", and an inAmount of "2,400.00". The price is "12.45000". The purpose of parsing the text (the electronic report shown in Figure 1) is to extract the data from the text.

The text description defines the document as a regular expression (regular expression, Regexes) to describe the matching rules of each block and each data. According to the matching rule, each block can be decomposed from text, such as the electronic report shown in FIG. 1, and the data can be extracted from the smallest unit block. .

The loading module 11 is used to load a text description definition document and text to be parsed. In this embodiment, text is loaded into the array in rows, with one element of the array corresponding to one line of text. Assuming that the electronic report in Figure 1 has a total of 50 lines, the electronic report is loaded into the array string, and the elements of the array are string[0], string[1], string[2], ..., st ring[48], String[49] corresponds to the first row, the second row, the third row, the ..., the 49th row, and the 50th row of the electronic report.

The parsing module 12 is configured to match each block of the text according to the matching rule of each block in the document according to the text description, and extract the data from the smallest unit block. When matching each block of the text, if the block series table type matches the block list, the first block matching the matching rule is matched. The specific implementation method of the parsing module 12 matching each block of text is as follows: starting from the root block, if the matched block includes sub-blocks, the document is defined according to the text description, and the sub-blocks are matched in the block according to the text description. Until all blocks are matched, and the data is extracted from the smallest unit block. The text description definition document defines the matching rules of each block, and finds the sub-text (that is, a part of the text) that matches the matching rule according to the matching rule of a block, that is, the block of the text is obtained. For example, according to the matching rule of the root block, the sub-text corresponding to the root block matching rule is searched in the text, that is, the root block of the text is obtained. It should be noted that, in this embodiment, the text is loaded into the array by row, and thus the matching is performed in the array, and the matched blocks are in units of array elements, for example, the root block system string[1] is matched. ~string[49], which is the 2nd to 50th lines of the text. In Figure 1 For example, the electronic report is matched. First, the root block root is string[1]~string[49]; in string[1]~string[49], the sub-block of the root block is matched. Get the sub-block list table[0] (representing form 1) table[1] (representing form 2), table[0], table[1] are respectively string[1]~string[24], string[26]~ String[49]; continue to match the sub-blocks of table[0], table[1], for example, get the sub-block title, from, date, supplierId, of table[0] from string[1]~string[24] supplierName, form are respectively string[1], string[2], string[3], string[5], string[5], string[9]~string[24], and get the title "***有限公司Purchase order", from is "H5S00001", date is "20070601 17:32:03", supplierId is "9876543210", supplierName is "*** electronic company"; form block contains item sub-block, item series table type , then continue to match the item sub-block list item[0], item[1], item[2], item[3], item[4] in the block form (for example, string[9]~string[24]) , item[5], for example, get string[10], string[12], string[14], string[ 16], string[18], string[20], continue to match the child sub-block, such as matching the sub-block of item[0] in string[10], get consignId as "HaA-012345", consignDate as " 20070512", productId is "987654J00-001-BB", PoId is "Ord-111111", inAmount is "2,400.00", and price is "12.45000".

The output module 13 is configured to output the extracted data according to the text description definition document in a document format required by the user. In this embodiment, the extracted capital is output by using an XML document. For example, the electronic report in FIG. 1 is parsed, and the extracted data includes the title, from, date, supplierId, supplierName of each form, and consignId, consignDate, productId, PoId, inAmount, price of each record in the data table. The extracted data is output in the output.xml document according to the structure of the text. In an embodiment other than the embodiment, the output module 13 can also store the extracted data in a database by a field, for example, in an Excel table.

Referring to Figure 3, there is shown a flow chart of a preferred embodiment of the text parsing method of the present invention.

Step S301, establishing a text description definition document, the text description defining a structure of the document description text and a matching rule. In this embodiment, the text description defines that the document is an XML document, ie, *.xml.

The text description defines a block in which the document defines a plurality of hierarchical tree structures to describe the structure of the text. The top layer block is the root block, and the root block contains several blocks, each of which contains several blocks, thereby finally forming a multi-layered tree structure. For any non-root block, only one block directly contains the non-root block (called a sub-block). If a block contains multiple sub-blocks of the same structure, the sub-block family table type is defined. The bottom block does not contain sub-blocks, the bottom block is called the smallest unit block, and the smallest unit block contains only data. For example, for the electronic report in FIG. 1, the root block is represented by root, corresponding to the entire electronic report, and the sub-block of the root block is represented by a table, and the table is defined as a list type, corresponding to each form in the electronic report, and the table includes The sub-block title, from, date, supplierId, supplierName, and form respectively indicate the title, source, tabulation date, vendor number, vendor name, and data table of each form; form includes item sub-block, and item is defined as list type. , corresponding to each record in the data table; item includes sub-block consignId, consignDate, productId, PoId, inAmount, and price, respectively, indicate the receipt number, receipt date, item number, purchase order number, receipt quantity, and receipt unit price for each record. The title, from, date, supplierId, supplierName, consignId, consignDate, productId, PoId, inAmount, and price are the smallest unit blocks, and the blocks contain data. For example, in Form 1 of Figure 1, title is "*** Ltd. Purchase Order", from "H5S00001", date is "20070601 17:32:03", supplierId is "9876543210", supplierName is "*** Electronic "Company", the first record in Form 1 has a consignId of "HaA-012345", a consignDate of "20070512", a productId of "987654J00-001-BB", a PoId of "Ord-111111", and an inAmount of "2,400.00". The price is "12.45000". The purpose of parsing the text (the electronic report shown in Figure 1) is to extract the data from the text.

The text description definition document uses a regular expression to describe the matching rules of each block and each material. According to the matching rule, each block can be decomposed from text, such as the electronic report shown in FIG. 1, and can be from the smallest unit area. The data is extracted from the block.

In step S302, the loading module 11 loads the text description definition document, as in the embodiment, the *.xml file is loaded.

In step S303, the loading module 11 loads the text. In this embodiment, text is loaded into the array in rows, one element of the array corresponding to one line of text. Assuming that the electronic report in Figure 1 has a total of 50 lines, the electronic report is loaded into the array string, and the elements of the array are string[0], string[1], string[2], ..., string[48], String[49] corresponds to the first row, the second row, the third row, the ..., the 49th row, and the 50th row of the electronic report.

Step S304, the parsing module 12 defines each block of the matching text according to the matching rule of each block in the document according to the text description, and extracts the data from the smallest unit block. When matching each block of the text, if the block series table type matches the block list, the first block matching the matching rule is matched. The specific implementation method of the step is as follows: starting from the root block, if the matched block includes the sub-block, the document is defined according to the text description, and each sub-block is matched in the block until all the blocks are matched. And extract data from the smallest unit block. The text description definition document defines the matching rules of each block, and finds the sub-text (that is, a part of the text) that matches the matching rule according to the matching rule of a block, that is, the block of the text is obtained. For example, according to the matching rule of the root block, the sub-text corresponding to the root block matching rule is searched in the text, that is, the root block of the text is obtained. It should be noted that, in this embodiment, the text is loaded into the array by row, and thus the matching is performed in the array, and the matched blocks are in units of array elements, for example, the root block system string[1] is matched. ~string[49], which is the 2nd to 50th lines of the text. Taking the electronic report in Figure 1 as an example, the electronic report is matched. First, the root block root is string[1]~string[49]; in the string[1]~string[49], the root block is matched. Sub-block, get the sub-block list table[0] (representing form 1) table[1] (representing form 2), table[0], table[1] are respectively string[1]~string[24], string [26]~string[49]; continue to match the sub-blocks of table[0], table[1], for example, get the sub-block title of table[0] from string[1]~string[24], from, Date, supplierId, supplierName, form are respectively string[1], string[2], string[3], string[5], string[5], string[9]~string[24], and get the title "***有限公司Purchase order", from is "H5S00001", date is "20070601 17:32:03", supplierId is "9876543210", supplierName is "*** electronic company"; form block contains item sub-block, item series table type , then continue to match the item sub-block list item[0], item[1], item[2], item[3], item[4] in the block form (for example, string[9]~string[24]) , item[5], for example, get string[10], string[12], string[14], string[16], string[18], string[20], continue to match the child sub-block, as in string Matching the sub-block of item[0] in [10], the consignId is “HaA-012345”, the consignDate is “20070512”, the productId is “987654J00-001-BB”, the PoId is “Ord-111111”, and the inAmount is “ 2,400.00", the price is "12.45000".

In step S305, the output module 13 outputs the extracted data according to the text description definition document in a document format required by the user. In this embodiment, the extracted data is output by using an XML document, for example, the electronic report in FIG. 1 is parsed, and the extracted data includes title, from, date, supplierId, supplierName, and each record in the data table of each form. consignId, consignDate, productId, PoId, inAmount, price, the extracted data is output in the output.xml document according to the structure of the text. In an embodiment other than the embodiment, the output module 13 can also store the extracted data in a database by a field, for example, in an Excel table.

According to the above method, for various types of text, even if the text is complicated in format and large in volume, the text description can define the structure of the document description, and the document can be extracted according to the text description definition document.

The text analysis system and method of the present invention are disclosed above in the preferred embodiments, and are not intended to limit the present invention. Any person skilled in the art will be able to make changes and refinements without departing from the spirit and scope of the invention, and the scope of the invention is defined by the scope of the appended claims.

S301‧‧‧Create text description definition document

S302‧‧‧Load text description definition document

S303‧‧‧Load text

S304‧‧‧Extracting information

S305‧‧‧Output data

Claims (6)

A text parsing system, the system comprising: a loading module for loading a text description definition document and loading the text to be parsed into an array, one element of the array corresponding to one line of text, the text description definition The document defines the various blocks of the text and the matching rules of each data. Each block forms a multi-layered tree structure, the top layer is the root block, and the bottom layer is the smallest unit block; the parsing module is used to start from the root block, according to Matching rules of the respective blocks match the respective blocks of the text in the array, and if the matched blocks include the sub-blocks, the sub-blocks are matched in the matched blocks according to the matching rule of the sub-blocks The block, until all the blocks are matched, and each piece of data is extracted from each of the smallest unit blocks according to the matching rule of the corresponding data; and an output module is configured to output the extracted data in a document format required by the user. The text parsing system according to claim 1, wherein the text description definition document is an extensible markup language document, and the text description defines a matching rule in which the document uses a regular expression to describe each block and each material. The text parsing system according to claim 1, wherein the output module organizes the extracted data in a document format required by the user, and organizes the extracted data according to the text description definition document into an extensible markup language. Document output. A text parsing method, the method comprising the steps of: creating a text description definition document, wherein the text description defines a matching rule for defining each block and each material of the document, each block forming a multi-layered tree structure, and the top layer is a root block, The bottom layer is the smallest unit block; Loading the text description defines the document and loads the text to be parsed into the array. One element of the array corresponds to one line of text; starting from the root block, each of the texts is matched in the array according to the matching rule of each block a block, if the matched block includes a sub-block, the sub-block is matched in the matched block according to a matching rule of the sub-block until all the blocks are matched, and according to the corresponding data The matching rule extracts each piece of data from each of the smallest unit blocks; and outputs the extracted data in a document format required by the user. The text parsing method according to claim 4, wherein the text description defining document is an extensible markup language document, and the text description defines a matching rule in which the document uses a regular expression to describe each block and each material. The text parsing method according to claim 4, wherein the extracted data is output in a document format required by the user, and the extracted data is organized into an extensible markup language document according to the text description definition document.