TW200535641A - Online extraction rule analysis for semi-structured documents - Google Patents

Abstract

The present invention provides an efficient automated method for online information extraction. The proposed extractor OLERA converts semi-structured documents into structured data as a user encloses, an interesting block from one of the training pages via hierarchy page encoding, approximate pattern matching and multi-string alignment. Finally, a "specify" procedure assigns the schema of the extraction target for extraction rule generation.

Description

200535641 V. Description of the invention (1) The technology to which the invention belongs The invention is based on a semi-structured prior art. With the Internet Display Language (ΗTML), how users can present information through information acquisition. The important design information of the manual writing and extraction format is generated at any time. The extraction program is used to generate extraction rules from 1997. It is related to an information extraction method, and especially an online information extraction method of a document. . The grid of the Internet can be retrieved and then the topic of dynamic data retrieval may be set to start the development of the web page and then presented more and more. It is useful and spends a lot of time in the design of the system to integrate the data. Construct a wrapper program that improves the search efficiency and the most direct method of the system (Wrapp changes, so if the capture system is the largest, Wrapper Induct, tells the system to use the rules to capture the network N. Hsu and C .-C. Chang. Finite

for semi-str of I JCAI-99 Techniques a Stockholm, S Generating F uctured text mining. Workshop οn Text min nd Applications, p p. wed en, 1 9 9 9; C. -N. H inite-state transduc Superimpose useless information with hypertext to find it. Therefore, the rich and poor materials are structured by rich search engines, so that the accuracy is based on the use of online resources for various websites. However, due to how fast and automatically the websites challenge. The method of i ο η is proposed, the information that makes the system poor (see C. state transducers In Proceedings ing: Foundations, 38-49, s u and Dung, e r s for < 1

12652TWF.PTD Page 5 200535641 V. Description of the invention (2) semi-structured data extraction from the web. Information Systems, 23 (8): 521-538, 1998; N.

Kushmerick, D. Weld, and R. Doorenbos. Wrapper induction for information extraction. In Proceedings of the 15th International Joint Conference on Artificial Intelligence (IJCAI), pp. 7 2 9-7 3 7, Japan, 1 9 9 7; I Muslea, S. Minton, and C. K η o b1o c k. A hierarchical approach to wrapper induction. In Proceedings of the 3rd International Conference on Autonomous Agents, pp. 1 9 0-1 9 7, S ea 11 1 e , WA, 1 9 9 9) This kind of method of labeling example websites and pages (called supervised approaches), although has a good retrieval rate, must go through a very complicated labeling to generate extraction rules. It is not so convenient for users, so how to reduce the information retrieval system marked by users is a major challenge in system design. At present, the method without user identification (called unsupervised approaches) includes 4 tongues such as IEPAD (please refer to C. —Η. Chang and S. -C. Lui. Iepad: Information extraction based on pattern discovery. In Proceedings of the 10th International Conference on World Wide Web, pages 6 8 1-6 8 8, Η ο ng-K ο ng, 2 0 0 1) It is assumed that there are multiple pieces of data in the document, so it can use repetitive patterns The prospecting method is used as a guess of possible objects, so there is no solution for a single record webpage. Another example is the existing RoadRunner (see V. Grescenzi, G.

12652TWF.PTD Page 6 200535641 V. Description of the invention (3) Mecca, and automatic In Proceed on Very La EXALG (please refer to Extracting Proceeding Conference 2 0 0 3.) It is hoped to open the whole, but you can also enter in the model If you still need to invent it, you need to use the method of both the invention and the invented method to complete the Extraction. The information field of the document of this invention can be recorded on the webpage. It also aims at P. Merialdo. Roadrunner: Towards data extraction from large web sites. , ings of 2 7th International Conference rge Data Base, pp. 109-118, 2 0 0 1 ·) and A. Arasu and H. Garcia-Molina. structured data from web pages. In s of ACM SIGMOD International on Management of Data, pages 337-348, Suppose that the captured data is based on the entire web page. The template (temp 1 ate) and data (data) of the web page can distinguish each person's required data, even if it is It can have the data it wants, so there is no need for the user to post-process the data needed by the user. And it takes the strength of the designer between supervised and unsupervised, and has good performance. With the above-mentioned prior art, the present invention proposes an effective and automatic information retrieval system (I nf 〇rmati ο η System), which allows users to obtain a complete collection without having to go through complicated labels, and simultaneously solve single records and multiple Pen capture issues. Provides an information manipulation method suitable for half-structured retrieval. The user selects the target area within the semi-structured document. The target information is analyzed in detail to make the semi-structured document structure.

I2652TWF.PTD Page 7 200535641 V. Description of the invention (4). Finally, specify attributes to improve the retrieval rate of the information retrieval method. The foregoing detailed analysis steps include: (1) drill down (dri 1 1-d 0w η), which is a detailed analysis of the data; (2) shanghe (r ο 1 1-u ρ), which integrates the data as Attributes are higher-level data; (3) Specifying attributes is to determine a schema and extract and integrate according to the schema. This method of information retrieval also implements the above steps in conjunction with the methods of hierarchical coding, approximate pattern matching, and string alignment. Among them, the steps of approximate pattern matching are implemented in a similar string alignment manner (see D. G usfie 1 d · E ificient methods for multiple sequence alignment with guaranteed error bounds. Bull. Math. Biol ·, 5 5: 1 4 1-1 5 4, 1 9 9 3 ·). The string alignment step is implemented by, for example, multi- # string alignment. The target information is recorded in, for example, Hypertext Markup Language (HTML). The target information may be a single record page or multiple record pages. On the other hand, the step of analyzing the extraction rules further includes (1) taking the definition symbol as the basis (Del im iter-based), relying on the delimitation symbol, to retrieve according to the context format of the attribute to be retrieved; (2) taking The content is based on (C ο ntent-based), which is retrieved based on the data content; (3) The above two methods are used together. The result of the information retrieval method in the present invention is stored in an Extensible Markup Language (XML) format, so it can be used with XML-related applications. In addition, when the target _ area is framed, for example, a multi-frame selection operation is performed to solve the problem of irregular arrangement of the attribute ′ in the web page. The invention also provides a method for generating information extraction rules, which is applicable to

12652TWF.PTD Page 8 200535641 V. Description of the invention (5) Training on one or more record pages to generate extraction rules. This method for generating information extraction rules includes a user selecting a target area within the record pages, which includes at least one target information, and makes detailed data designation for the target information. The method for generating the extraction rules described herein includes detailed analysis steps including: (a) drill-down, which performs a detailed analysis of the target data; (b) r ο Π-up ), Which integrates the target data into higher-level data; and (3) Specifying attributes, determine a schema and extract integration based on the schema. The method for generating information extraction rules of the present invention, when implemented, further includes the steps of pattern comparison, string alignment, and comparison of adding plain text content. On the other hand, the aforementioned down-step further extracts the attribute value based on the demarcation mark. The closing step is integrated with a part of the blocks in the target area. The final attribute assignment step is completed by the user clicking the mouse and entering the attribute name. The invention also provides a method for generating information retrieval rules to process one or more record page trainings to generate information retrieval rules. The record pages are first processed into a structured document, and then detailed data assignments are made for the structured document to retrieve attribute values. In the present invention, structured documents are first generated by processes such as hierarchical coding, approximate pattern matching, and string alignment. Then make detailed data designation for the structured document to retrieve the attribute values. The steps include (a) drill down (dri 1 1-d 0w η), which is a detailed analysis of the target data; (b) shanghai ( r ο 1 1-u ρ), is to integrate the target data as higher-level data; (3) Specify the attributes, determine a schema, and perform integration according to the schema.

12652TWF.PTD Page 9 200535641 V. Description of the invention (6) In order to make the text obvious and easy to understand, the detailed description of the embodiment of the present invention extraction complex labeling mouse click attribute labeling simplified The above and other objectives, features, and advantages can be further exemplified below with a preferred embodiment, and in accordance with the accompanying drawings, as follows: The proposed online extraction rule analysis (OLA) system is mainly aimed at Avoid letting the user only use a few simple, WYSIWYG (what you see is what you get) concepts to reduce the complexity into a three-step operation during the retrieval process. (1) Use the target area to select) to allow users The target to be retrieved uses a simple mouse selection to make the system intuitively record other possible data in a smart way; (2) Drill-down or Roll-up of the block According to the record of the first step, the data acquisition, alignment (Speci extraction system or multiple mechanized general domains) and the use of important data to show the author's interest are approximated by Pattern matching and string A 1 ignment) to establish mining rules; (3) the relevant attribute refers to fy), which uses the comparison of the addition of pure text content to enhance the overall success rate 'regardless of the single record page ( Singular Page) Multi-Record Page, the results have completed their self-imagination, and also have a high capture rate. For example, a webpage can divide its content into areas that the user is interested in, and areas that the user is not interested in, and the area that the user is interested in is the display place. Take the webpage that introduces books as an example, information such as, publisher, publication date, etc. may be displayed; the user's unseen area is the buttons or links on the website that are easy for the viewer to operate.

12652TWF.PTD Page 10 200535641 V. Description of the invention (7) For the various users, the required information is actually different, so the user must tell the system what he thinks and feels through the box selection steps. Interesting area. After the target area is selected here, the system's processing method is mainly through an improved alignment (a 1 i g n m e n t) algorithm. This alignment algorithm is described here. First of all, the content selected in the target area is web-encoded. Because the target is constructed in Η TM L format, the HTML tag is used as the delimiter when interpreting the content of this webpage, and between two adjacent definition symbols , The content is ignored, so the content between two adjacent definition symbols is defined as a unit. In order to execute the alignment algorithm used in the present invention, the above-mentioned web page coding method is used to hierarchically encode the content selected in the target area. For example, hierarchical coding can usually be divided into three levels according to the size of the coverage: review level (markup-level), text level (text-level), and character level (w 〇 r d-1 e v e 1). The meaning of these three levels can be understood from the table below. It should be stated that the list is only for the purpose of example. All definition symbols at the same level can be distinguished by similar level coding methods. At this point, a matching pattern is generated, which can also be referred to as a reference acquisition rule generated in the training process, for subsequent data testing and acquisition.

12652TWF.PTD Page 11 200535641 V. Description of the invention (8) Hierarchical coding scheme definition mark review [^ block-level tags block-level tags text (text) paragraphs NL, CR, Tab sentence孑.?! Word phrase:,; [] () {} _ Blank valve $-/ After hierarchical coding, approximate matching is performed, which is performed by comparing blocks in the selection box on the training webpage. Similar records. Those that have a similarity to the box-selected block that are less than a critical level meet the recognition criteria. The content can be recorded through the approximately matched web pages, and then can be processed by multiple string alignment algorithm. This algorithm uses a dynamic program to execute a total of nk times to obtain the alignment result, which is calculated for k strings of length n. For example, for the following three words B1, B2, B3, the following alignment results are obtained after executing this algorithm:

This method has been applied to I EPAD, which can make up for the shortcomings of the PAT tree algorithm, and is especially suitable for situations with similar word rate lengths. The number of strings processed is very large. You can use the known c e n t e r s t a r (see D. G u s f i e 1 d · Efficient methods for multiple sequence alignment

12652TWF.PTD Page 12 200535641 V. Description of the invention (9) with guaranteed error 55: 141-154, 1993 ·) Approximately to this point, at the beginning, when the user framework converts it into a structure, it is necessary to further correctly extract the reference, It may be the price, a structure that is processed and converted to obtain these attribute values. After all, the current markup language (HTML) is the definition, which means that at most the word content can only contain the text content that contains the attributes you want to retrieve. Extraction, which can be divided into Roll-up and Specify, Dril bounds. Bull. Math. Biol., Algorithms to speed up the calculation. The input web page has passed the system's information. Nonetheless, the present invention also has a definite objective. The author's name or date of an attribute value can be used to change the information, which does not mean that all methods have been taken so far. : er) based on the assumption that a paragraph of text content is taken out, and this paragraph value, so it must be further specified for the purpose specified in detail. It is divided into three parts. The main ideas of Drill-down, 1-down and Roll-up come from the 0LAP multi-dimensional data model. The standard 0LAP also includes Drill-down and Ro11-up. Among them, Drill-down is for data. Then {more detailed analysis, R ο 1 1-up is a higher-level concept of integrated data, the same reasoning is also specified in the detailed data, D ri 1 1-d ωw η will change the content of pure text For detailed analysis, the pure text content is separated by some punctuation marks or line breaks as De e imiter to further extract the attribute values. Ro 1 1 -up integrates the data of a certain block, like The author's data is integrated into the author list or the publication date of the book and the publisher's data are integrated into the publication-related data, etc. This part can be performed according to the needs of the user. Whether it is D r i 1 1-d〇w η or R ο 1 1-u p,

12652TWF.PTD Page 13 200535641 V. Description of the invention (10) The next two steps are implemented: web page encoding and multiple string alignment algorithm. These two steps are the same as those described in the block selection step, and are not described here. 0

Drill-down and Roll-up provide a very flexible way for users to retrieve attribute values. Then there is another function Spec i fy, S pecify is an important task to determine the outline (S chema). After the outline is determined, Extract and integrate according to this outline. In the conventional information retrieval system, the outline is usually determined at the beginning, and in the 0LERA system, it is the last step, and it only needs a few mouse clicks and the name of the attribute name can be completed. In the system proposed by the present invention, from the viewpoint of What You See Is What You Get (WYSIWYG), the complicated attribute labeling is simplified into a three-step simple operation in sequence. Please refer to FIG. 1, which illustrates a target area selection method according to a preferred embodiment of the present invention. The user clicks and drags with a simple mouse, or through other user-interfaces with the same functions (such as a mouse pad), to determine the content to be retrieved in the landing page. The content includes, for example, plain text, and also includes, for example, images Like, and it is expressed in Hypertext Markup Language (HTML) format. In Figure 1, the website of the BARNES & N 0 BLE online bookstore is taken as an example. In the searched J + + reference book web page, the user selects the interest The content includes the title, author, format, International Standard Book Number (ISBN), publisher, publisher, publication date, edition description, etc. (102) , And the unselected part of this page includes a text statement to the right of the layout

12652TWF.PTD Page 14 200535641 V. Description of the Invention (11) (104) ° Next, referring to FIG. 2, a schematic diagram of an interactive interface with detailed information according to a preferred embodiment of the present invention is shown. After the OLERA system performs web page encoding, approximate pattern matching, and multiple string alignment calculations, the data in the form of table 2 1 0 shown in FIG. 2 is obtained, and the shaded part 2 2 0 in the figure is 0ELRA The retrieval rules generated after the calculation, other similar book query web pages can structure the data in the selected block according to this rule. To implement data retrieval, the following retrieval rules need to be used. The representation of the extraction rules can be roughly divided into three types. (1) Based on the definition symbol (D e 1 i in iter-based): With the delimitation symbol, it is extracted according to the context format of the attribute to be retrieved, such as the figure As shown in Fig. 3, the symbols 〃 $, · ", etc. entered in the pop-up interactive window 3 3 0; (2) Based on the content (C ο ntent-based). Special formats such as book title, author, layout, publisher, international standard book number, publication date, version, etc. in the preferred embodiment; (3) Context-based: (1), ( 2) Both methods. Figure 3 shows an example of performing a drilldown (D ri 1 1-d 〇w η). In this embodiment, a drilldown is performed for a column containing format (f ommat) information. According to the delimiter symbol typed in the interactive window 3 3 0 (3 symbols are typed here, but the comma ", M is used), the layout data is reduced to π and then parsed into n F 〇rmat: π, " P aperbac kπ, π 1 sted · π, '' 9 9 2 ρ ρ · π and other four columns to obtain the data in the form of a table 3 1 0. Referring to FIG. 4, an embodiment of description is described. The operation can be regarded as

12652TWF.PTD Page 15 200535641 V. Description of the invention (12) The reverse operation, which means to combine the subdivided data in several columns into a higher-level data, such as the shaded part 4 2 0 in Figure 4, the Data such as " Paperback π, π First Edition π, M Total Pages π, etc. are collected into π layout π — pen data. The user can fill in the desired separator symbol in 4 3 0 according to his needs. Next, referring to FIG. 5, the related attribute designation (S p e c i f y) is described. As shown in FIG. 5, the user enters "Book Title11," Author ", and π P u b 1 i s h e r n is on the desired data column. Referring to FIG. 6 ′, it is not intended to illustrate the flow of the OLED system architecture of the present invention. According to the diagram, the operation method of the 0 L E R A system is divided into two parts: the training program 6 1 0 shown in the left half and the test / retrieval program 6 2 0 shown in the right half. Among them, in the training process, a template file 602 is, for example, a single recording webpage or a multiple recording webpage. First, it is retrieved through the user operation block selection 612, and then it is retrieved through the steps of drill down / up and down 614, and attribute specification 616 Pattern 6 0 4. In the test / retrieval procedure 620, the target file is processed in step 626, and the extracted pattern 604 data is introduced. It should be noted that in this flowchart, all the steps marked with an oval are user-operated, such as clicking and dragging with a mouse or performing similar effects with other interfaces, and all the steps marked with a rectangle are the LELER system of the present invention. The corresponding operation. When the user selects a block (step 6 1 2), the corresponding step 6 2 2 performed by the system includes recording page coding, approximate pattern matching, and multiple string alignment. The system operation corresponding to the user's down / close step (6 1 4) includes recording page encoding and multiple string alignment. On the other hand, in the test / retrieval procedure steps, the test target document or

12652TWF.PTD Page 16 200535641 V. Description of the invention (13) Retrieving the target file 6 0 6 The structured data 6 0 8 is obtained through the steps including recording page encoding and multiple string alignment. Although the present invention has been disclosed as above with a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can make some changes and retouch without departing from the spirit and scope of the present invention. The scope of protection of the invention shall be determined by the scope of the attached patent application.

12652TWF.PTD Page 17 200535641 Brief Description of Drawings Figure 1 is a schematic diagram of a method for selecting a target area according to a preferred embodiment of the present invention. FIG. 2 is a schematic diagram of a detailed data specifying interactive interface according to a preferred embodiment of the present invention. FIG. 3 is a schematic diagram of an interface for detailed analysis of data drilling (D r i 1 1-doww η) according to a preferred embodiment of the present invention. FIG. 4 is a schematic diagram of a Roll-up interface for integrating data into a higher level according to a preferred embodiment of the present invention. FIG. 5 is a detailed data specification according to a preferred embodiment of the present invention. Interactive interface diagram. FIG. 6 is a schematic diagram of a data retrieval process of the 0LERA system according to a preferred embodiment of the present invention. Description of Schematic Symbols 602 Files 604 Extraction Patterns 606 Files 608 Structured Information 610 Training Procedures 612 Block Selection 6 14 Drilldown / Closing 616 Designation Properties 620 Test / Retrieval

12652TWF.PTD Page 18

Claims (1)

200535641 6. Scope of patent application 1. A method for generating information extraction rules, which is suitable for processing one or more record pages, including: The user selects a target area within the record pages, including at least one target information; The target information specifies a detailed data to generate a structured data; and an information retrieval rule is generated based on the structured data to retrieve a subsequent record page. 2. The method for generating information extraction rules as described in item 1 of the scope of patent application, wherein the steps of specifying the detailed data include: drill down (dri 1 1-d 〇w η), which is a detailed analysis of the target data ; Shanghe (r ο 1 1-u ρ), is to integrate the target data as a poor material with higher attributes, and to specify the attributes, determine a schema and extract integration based on the schema. 3. The method for generating information extraction rules as described in item 1 of the scope of patent application, further including the steps of pattern comparison, string alignment, and adding plain text content comparison. 4. The method for generating information extraction rules as described in item 1 of the scope of patent application, wherein the step of drilling down further includes extracting attribute values based on a demarcation mark. > 5. The method for generating information extraction rules as described in item 1 of the scope of patent application, wherein the step of closing includes using a part of the block in the target area 12652TWF.FTD Page 19 200535641 6. Scope of patent application for integration. 6. The method for generating information extraction rules as described in item 1 of the scope of patent application, wherein the steps of specifying attributes include a user clicking a mouse and entering plain text data. 7. The information extraction rule generation method described in item 1 of the scope of patent application, wherein the step of specifying attributes is the last step. 8. A method for generating information extraction rules, which is suitable for processing one or more record pages, including: processing the record pages into a '^ structured document, and specifying a detailed data for the structured document for retrieval Property value. And 9. The method of generating information extraction rules as described in item 8 of the scope of the patent application, wherein the steps of processing the record pages into the structured document include: hierarchical coding; approximate pattern matching; and string alignment. 1 0. The method for generating information retrieval rules as described in item 8 of the scope of the patent application, wherein the step of specifying detailed data for the structured document to retrieve the attribute values includes: drill down (dri 1 1-d 〇w η), to perform a detailed analysis of the target data; • 'Shanghe (r ο Π-u ρ), to integrate the target data into a poorer material with higher attributes, and I2652TWF.PTD Page 20 200535641 VI. Scope of patent application The designated attribute is to determine an outline (s c h e m a) and extract and integrate according to the outline. 1 1. A method for generating information extraction rules, comprising: a user selecting a target area on a record page: automatically dividing the content of the target area into a plurality of data parts according to a flag in the target area; Display the data sections; the user decides how to process the data sections to obtain the desired result; and according to the user's decision process, generates a corresponding information retrieval rule for capturing subsequent record pages. 1 2. The method for generating information extraction rules as described in item 11 of the scope of patent application, wherein the user decides that the processing method of these data parts includes at least one of closing, dropping, and specifying attributes. 1 3. A method for generating information extraction rules, including: selecting a target area using a frame on a record page: generating a user window interface to provide options for processing the content of the target area; and according to the user window interface The selection process of the options generates corresponding information extraction rules. 14. The method for generating information extraction rules as described in item 13 of the scope of patent application, wherein the options for processing the content of the target area include at least one of closing, drilling down, and specifying attributes. 1 5. Generate information extraction rules as described in item 13 of the scope of patent application 12652TWF.PTD Page 21 200535641 VI. Patent Application Method, wherein the selection process of the user window interface includes selecting a delimitation symbol as a basis for processing the content of the target area. 16. The method for generating information extraction rules as described in item 13 of the scope of patent application, wherein the selection process of the user window interface includes the user inputting a plain text string as a basis for processing the content of the target area . 12652TWF.PTD Page 22