KR20020084944A - Automatic information extraction method from a web text using mDTD rule - Google Patents

Abstract

PURPOSE: A method for automatically extracting the information of a web document using an mDTD(modified Document Type Definition) grammar rule is provided to conveniently and efficiently extract many information from the vast information of a domain by using the mDTD rule through the mechanical repetition learning. CONSTITUTION: The method for the mechanical learning comprises the steps of collecting the web document from the domain(S1), transforming the web document into a text object(S2), extracting a sample data from the text object according to a previously written seed mDTD rule(S3), attaching a format element tag to the sample data(S4), and generating the proper mDTD rule by using the tagged sample data(S5). The method for the automatic extraction comprises the steps of collecting the web document from the domain(S11), transforming the web document into the text object(S12), attaching the format element tag to the text object(S13), extracting a target by judging which mDTD rule among the mDTD rules generated by the mechanical learning process is suitable for the tagged text object(S14), and storing the extracted target in a domain database(S15).

Description

Automatic information extraction method from a web text using mDTD rule}

The present invention relates to a method for automatically extracting web document information, and more particularly, to a method for automatically extracting web document information using a simple mDTD grammar rule.

Recently, a myriad of information is provided on a website, such as home appliances, travel information, financial information, and the like. Such information is collected by a professional information provider and made into a database in order to provide a user with specific information.

In such a situation, the method of extracting a large amount of information from the Internet domain in order to make a database has a complicated structure and is made by a highly skilled technician, and much of the time is required for extracting necessary information because of the manual work. There was a problem.

Moreover, as the user's preferences are diversified, the suppliers of various service products, including manufactured products, tend to build specialized databases. Therefore, in order to extract information to be stored in a database in a variety of ways quickly and flexibly, there is a need for a method that can extract much more information more efficiently and conveniently than before.

An object of the present invention for solving the above problems is to define a simple mDTD grammar and to create a simple seed mDTD rule for a specific domain by the mDTD grammar in order to efficiently and automatically extract a large amount of information of the domain, An easy-to-use mDTD that extracts examples from the domain by seed mDTD rules and generates learned mDTD rules through iterative machine learning, extracts the necessary data from the domain by the learned mDTD rules, and then databases Provides an automatic web document information extraction method using grammar rules.

In addition, another object of the present invention to build a database that can support a data service system of various applications such as e-commerce comparison shopping and product search, web document automatic information extraction using the mDTD grammar rules according to the above method To provide a system.

1 is a diagram illustrating a definition of a modified document type definition (mDTD) grammar according to the present invention.

2 illustrates a seed mDTD rule for an audio and visual product in accordance with an embodiment of the present invention.

3 illustrates an algorithm of a machine learning methodology in accordance with an embodiment of the present invention.

4 is a diagram illustrating an mDTD rule in which an example of an AV product extracted using the seed mDTD rule has been machine-learned according to the algorithm of FIG. 3.

5 is a block diagram of a web document automatic information extraction system using mDTD grammar rules according to an embodiment of the present invention.

6 is a flowchart of a method for automatically extracting web document information using mDTD grammar rules according to an embodiment of the present invention.

In order to achieve the above object, the present invention provides a web document automatic information extraction method using mDTD grammar, including a machine learning method and an automatic extraction method, the machine learning method is: (a) a web document from a domain Collecting; (b) a web document preprocessing step of converting the collected web document into a text object; (c) extracting example data from the preprocessed text object according to a pre-written seed mDTD rule; (d) a tag attaching step of attaching a stem tag to the extracted example data; And (e) machine learning to generate an mDTD rule appropriate for the domain using the tagged example data, the automatic extraction method comprising: (f) collecting a web document from the domain; (g) a web document preprocessing step of converting the collected web document into a text object; (h) a tag attaching step of attaching a stem tag to the preprocessed text object; (i) an mDTD grammar rule applying step of extracting a target by determining which mDTD rule is best suited among the mDTD rules generated in the machine learning step; And (j) storing the extracted target in a domain database.

In addition, another aspect of the present invention is to generate an mDTD rule representing the most examples from the sample data extracted in the machine learning step, and repeatedly performing a series of processes to delete the example until all the examples are processed. It features.

In addition, another aspect of the present invention provides a web document automatic information extraction system using mDTD grammar rules including (a) a learner, (b) an automatic extractor and (c) a domain database storage means, wherein the learner is configured from a domain. Means for collecting web documents; A web document preprocessor for converting the collected web document into a text object; An example extractor for extracting example data from the preprocessed text object according to a pre-written seed mDTD rule; A tag attacher attaching a stem tag to the extracted example data; And a machine learner that uses the tagged example data to generate an mDTD rule appropriate for the domain, wherein the automatic extractor uses the learner's collection means, a web document preprocessor and a tagger to generate a web document from the domain. And a mDTD grammar application means for extracting a target by determining whether a tagged text object generated by collecting, preprocessing, and tagging is most suitable with which mDTD rule among the mDTD rules generated by the machine learner, and extracting a target; The extraction target is stored in a storage means.

Hereinafter, with reference to the accompanying drawings will be described in detail the present invention.

Figure 1 shows a grammatical form named mDTD defined to extract only the necessary information from the vast amount of information on the Internet domain. Here, the Internet domain to be extracted information refers to information about a specific field published on the Internet, such as home appliances, travel information, financial information, personal information, etc., and to extract information from a specific domain. This is called the extraction target. That is, for example, personal information information, interests such as name, address, date of birth, occupation, etc. are targeted for extraction.

The mDTD grammar shown in FIG. 1 serves as a programming language that optimally represents a domain, and has a declarative characteristic when defined as a rewriting rule. The mDTD grammar definition is described in more detail below. The first line represents the basic form of mDTD grammar rules. A rule consists of a start indicator (<!) Indicating the start of the rule, a keyword, a rule name, opt, (content), occurrence_op, action, and a rule end indicator (>) indicating the end of the rule.

As defined formally in FIG. 1, the keyword is one of a TARGET representing an extraction target, an ELEMENT representing an element, an INSTANCE representing an instance object, and an ENTITY representing lexical data.

For reference, the DTD grammar that is the basis of the mDTD grammar described above is defined in the Internet site http://www.w3.org/xml/1998/06/xmlspec-report.

FIG. 2 illustrates a seed mDTD rule written for an AV (Audio and Visual product) domain on the Internet using the mDTD grammar defined as shown in FIG. 1. As shown in FIG. 2, the seed mDTD rule is composed of a rule for extracting examples for machine learning from a html document in a table form and rules for extracting information about an actual TV product. The seed mDTD rule must be written by the operator, but the seed mDTD rule can automatically obtain examples from the structural document, and through machine learning, which will be described later, such a basic framework applies equally to all domains. Can be.

The methodology utilizing such seed mDTD rules is a general methodology that can be extended to almost any domain on the Internet, which is described in more detail below.

3 shows the algorithm of the machine learning. The machine learning is to have the ability to process a larger number of sites belonging to the domain, and is used to extract almost all web site information belonging to the domain by extending the seed mDTD rule as shown in FIG. Create an mDTD rule. The machine learning method uses a sequential learning method. This is a method of generating a rule repeatedly matching the most examples among the input examples. Hereinafter, the algorithm of the machine learning will be described step by step with reference to FIG.

First, using a seed mDTD rule written by the operator and the above rules, one rule representing the most examples is generated from the examples extracted from the table structured Internet structure document. Then delete the examples from the entire example, and then again, a series of finding and deleting representative rules is repeated until all the examples have been processed. Here, the criterion adopted as a judgment material when determining whether a rule includes one example is based on the identity of strings and the identity of grammatical morphemes. Applicants of such a machine learning method are SML (Sequential mDTD Learning). It is called).

4 illustrates mDTD rules learned through the SML algorithm. The learned result rule is added to the seed mDTD rule. The added method creates a connection node (e.g., a rule such as mInst1 shown in FIG. 4), and the actual learned result node is located in the child node of the connection node. The reason for setting the connection node in the middle when adding the learned node is to prevent the direct connection of multiple nodes to one node. In addition, since the necessity of learning according to the extraction target is described differently when creating the seed mDTD, the learned results are learned for each extraction target.

4 is an example showing some results of learning about the model name, product name, manufacturer, and product characteristics of the AV domain, but in general, since the information required by the user is various and detailed, in order to obtain various extraction targets. There are hundreds of rules learned for each domain.

5 is a block diagram of a system for automatically extracting information from the Internet using mDTD rules. The system is composed of a learner 10, an extractor 20 and a domain DB 30, the learner 10 is a web robot 11 for collecting web documents from the domain, converting the collected web document to a text object A web document preprocessor 12, an example extractor 13 for extracting an example from a preprocessed text object, a stemmer tag attacher 14 for attaching a stemming tag to the extracted example, and an SML learner 15.

The extractor 20 applies the mDTD rules learned by the web robot 21, the web document preprocessor 22 and the stemmer tagger 24, and the learner 10, which are the same as the components of the learner 10, to target the target. The mDTD rule applier 25 to extract, the target frame buffer 26 which stores an extraction target, and the DB entry generator 27 which converts the stored target into the data type of a database are comprised.

6 is a flowchart illustrating a method for automatically extracting web document information using an mDTD rule. Hereinafter, a method of automatically extracting web document information using an mDTD rule will be described with reference to FIGS. 5 and 6.

First, the mDTD rule machine learning method by the learner 10 begins in step 1, in which the web robot 11 downloads and collects a web document from an initial uniform resource locator (url) associated with a specific domain. As described above, since a specific domain is a general term for information about a specific field published on the Internet, there are a plurality of urls related to the specific field. Since the document downloaded as above may not only be encoded in the html tag, but there may also be some spacing errors, in step 2 the web document preprocessing corrects this error by the web document preprocessor 12 and refines it into a pure text object. Perform

In the next step 3, the example extractor 13 extracts example data from the preprocessed text object based on the seed mDTD rule pre-written by the operator. In the next step 4, the stemming tag attacher 14 attaches the Korean stemming tag to the extracted examples for use in the SML learner 15.

In the next step 5, the SML learner 15 learns sequentially by the algorithm shown in FIG. 3 using the tag information and the string itself information, and generates an mDTD rule that is optimal for the domain. In this way, the learning of the mDTD rules is completed, and the generated mDTD rules are input to the extractor 20 to extract the target from the domain.

Next, the automatic information extraction method by the extractor 20 begins at step 11 where the web robot 21 downloads and collects the web document. The type of document downloaded by the web robot 21 is much more diverse than the type of document downloaded by the learner 10. The web robot 11 of the learner 10 downloads only structured documents in the form of a table, but the web robot 21 also downloads general semi-structured documents related to the domain.

Next, since the web document preprocessor 22 and the stemming tag attacher 24 which are to process the string itself perform the same processing as the corresponding part of the learner 10, the web document preprocessing step (step 12) and the stemming tag The attaching step (step 13) is carried out in the same manner as steps 2 and 4 and proceeds to step 14.

In the next step 14, the mDTD rule applicator 25 receives the mDTD rule machine-learned by the learner 10, and the text object input from the stemmer tag attacher 24 impersonates any of the learned mDTD rules. Determine if appropriate. In the next step 15, the determined object is stored in the extraction target frame buffer 26. The data stored in the target frame buffer 26 is treated only as one string data without distinguishing numbers or characters.

In the next step 16, the DB entry generator 27 converts the string data to match the data type of the database schema, and in step 17, the converted data is stored in the domain database 30, thereby automatically extracting information. Is complete.

Databases created in this way can have multiple uses. First, it can automatically build a domain database for each product and use it to provide backend services for systems such as price comparisons and product searches.

In addition, by describing various learning necessities according to extraction targets when creating a seed mDTD, the application field is infinite because the application field can be easily extended to various fields such as life, news articles, music, and movies.

According to the present invention, a simple mDTD grammar is defined, a simple seed mDTD rule is created for a specific domain by the mDTD grammar, and the examples are extracted from the domain by the created seed mDTD rule, and are learned through repetitive machine learning. By generating the generated mDTD rule, extracting the necessary data from the domain by the learned mDTD rule and storing it in the database, the necessary information can be efficiently extracted automatically from the domain.

In addition, according to the present invention, since the grammar used to extract data is provided as a simple mDTD grammar rule, the operator can easily use the grammar, and thus the application field is applied to various fields such as life, news articles, music, and movies. Can be easily extended, and thus the application field is infinite.

In addition, according to the present invention, if the operator simply writes the seed mDTD for a specific domain of the Internet, using the corresponding rule, for example, a product-specific domain database can be automatically built, and the price comparison, It can provide a backend service of a system such as a product search, and thus can support a data service system of various application fields such as e-commerce comparison shopping and product search.

Claims (11)

A web document automatic information extraction method using mDTD grammar including machine learning method and automatic extraction method, the machine learning method comprising: (a) collecting a web document from a domain; (b) a web document preprocessing step of converting the collected web document into a text object; (c) extracting example data from the preprocessed text object according to a pre-written seed mDTD rule; (d) a tag attaching step of attaching a stem tag to the extracted example data; And (e) machine learning using the tagged example data to create an mDTD rule appropriate for the domain, The automatic extraction method is: (f) collecting a web document from the domain; (g) a web document preprocessing step of converting the collected web document into a text object; (h) a tag attaching step of attaching a stem tag to the preprocessed text object; (i) applying an mDTD grammar rule for extracting a target by determining which mDTD rule is best suited among the mDTD rules generated in the machine learning step; And (j) a web document automatic information extraction method using mDTD grammar rules, comprising the step of storing the extracted target in a domain database. The method of claim 1, wherein step (e) generates an mDTD rule representing the most examples from the extracted example data, and repeats a series of processes of deleting the example until all the examples have been processed. Method for automatically extracting web document information using mDTD grammar rules. The mDTD according to any one of claims 1 to 2, wherein said steps (b) and (g) comprise correcting tag errors and spacing errors generated in said collected documents. Automatic information extraction of web documents using grammar rules. The method according to any one of claims 1 to 3, wherein step (a) and step (f) are performed by a web robot, and the type of web document collected in step (f) is determined by (a). Method for automatically extracting web document information using mDTD grammar rules, characterized in that it comprises a variety of semi-structured document than the form of the web document collected in the step. The method of claim 1, wherein after the step (i), storing the extracted target in an extraction target frame buffer (i ′) and extracting the extraction target stored in the frame buffer. And a database entry generating step (i ") of converting to a data type. Means for collecting web documents from the domain; A web document preprocessor for converting the collected web document into a text object; An example extractor for extracting example data from the preprocessed text object according to a pre-written seed mDTD rule; A tag attacher attaching a stem tag to the extracted example data; And A machine learner that uses the tagged example data to create an mDTD rule appropriate for the domain, The mDTD grammar rule is characterized in that the machine learner generates an mDTD rule representing the most examples from the extracted example data, and repeats a series of processes of deleting the example until all the examples have been processed. Learner for automatic web document information extraction. Means for collecting web documents from the domain; A web document preprocessor for converting the collected web document into a text object; A tag attacher attaching a stem tag to the preprocessed text object; MDTD grammar applying means for extracting a target by determining which mDTD rule is best suited among the pre-generated mDTD rules; A target frame buffer for storing the extracted target; Database entry generating means for converting an extraction target stored in the buffer to a data type of a database; And And a domain database storage means for storing the converted data. The extractor for automatic web document information extraction using mDTD grammar rules. An automated web document information extraction system using mDTD grammar rules including (a) learners, (b) automatic extractors, and (c) domain database storage means, The learner is: Means for collecting web documents from the domain; A web document preprocessor for converting the collected web document into a text object; An example extractor for extracting example data from the preprocessed text object according to a pre-written seed mDTD rule; A tag attacher attaching a stem tag to the extracted example data; And A machine learner that uses the tagged example data to create an mDTD rule appropriate for the domain, The automatic extractor is: The tagged text object created by collecting, preprocessing and tagging a web document from the domain using a collection means of the learner, a web document preprocessor and a tag attacher is configured with any mDTD rule of the mDTD rule created in the machine learner. MDTD grammar applying means for extracting a target by determining whether it is most suitable; The web document automatic information extraction system using mDTD grammar rules, characterized in that the extraction target is stored in the domain database storage means. The method of claim 8, wherein the machine learner generates an mDTD rule representing the most examples from the extracted example data, and repeats a series of processes of deleting the example until all the examples have been processed. Web document automatic information extraction system using mDTD grammar rules. 10. The extractor of claim 8 or 9, wherein the extractor is: Means for collecting web documents from the domain; A web document preprocessor for converting the collected web document into a text object; And And further comprising a tag attacher attaching a stem tag to the preprocessed text object, Web document automatic form using the mDTD grammar rules, characterized in that the form of the web document collected by the collection means of the automatic extractor includes a variety of semi-structured documents than the form of the web document collected by the learner collection means. Information extraction system. The method of claim 10, wherein the automatic extractor, A target frame buffer for storing the target extracted from the mDTD grammar applying means; And And a database entry generating means for converting the extraction target stored in the buffer into a data type of a database.