JP2020098592A - Method, device and storage medium of extracting web page content - Google Patents

Abstract

To provide method and device of extracting web page content.SOLUTION: The method of extracting web page content includes the steps of: calculating the similarity between a web page feature and a representative set of at least one web page feature cluster, the representative set including a sample of web page features that have a relatively high degree of similarity between each other in the corresponding web page feature cluster; determining a representative set that has the highest similarity to the web page features; updating the web page feature cluster associated with the determined representative set using web page features; recalculating the representative set of the updated web page feature clusters; and extracting the content from the web pages on the basis of the extraction template associated with the updated web page feature cluster.SELECTED DRAWING: Figure 2

Description

The present invention relates to a method, a device and a computer-readable storage medium for extracting web page content.

With the rapid development of the Internet, fixed access devices and mobile access terminals, web pages have become the main medium through which people obtain and generate information. However, as the number of web pages has increased exponentially, it is difficult to obtain necessary information quickly and accurately.

With the dramatic increase in digital resources and information on the Internet, there is a large amount of information that can be conveniently browsed by users. Therefore, it is also necessary to automatically extract the information that needs to be extracted according to the needs. Speaking of methods and systems for extracting web page content, it is usually about extracting data from semi-structured web documents, the core of which is implicitly contained in semi-structured HTML pages distributed over the network. Information points are extracted and expressed in a more structured and word-clarified form, which provides convenience for users to search data on the Web and for application programs to directly use the data on the Web. Especially. Since information extraction of web page contents is the first step in Internet information processing, the accuracy of information extraction can directly affect the subsequent processing. The purpose of information extraction is to extract noise and remove it, and obtain valuable information in a web page, for example, title, time, text, link, etc. of the web page.

Conventional web page information extraction methods include a rule-based extraction method and a machine learning-based extraction method. Although rule-based methods can achieve relatively high extraction accuracy, the process of making rules requires expert involvement. While it may be useful for small amounts of data because of the human involvement required, such human annotation methods cannot handle large amounts of data. Compared to rule-based extraction methods, machine learning-based methods do not require human involvement. However, such methods often require a large amount of annotated corpus. Moreover, since the creation of the annotated corpus needs to be completed manually, the conventional information extraction method based on machine learning also has locality.

It is an object of the present invention to provide a method, device and computer storage medium for extracting web page information. Compared with the prior art, the present invention can be used for processing huge amount of data, does not require a large amount of human annotation, and has higher accuracy, so that it can provide necessary information according to needs. It can be properly extracted.

To achieve the above object, according to one aspect of the present invention, there is provided a method of extracting web page content, which comprises a web page feature and a typical set of at least one web page feature cluster. And a representative set includes samples of web page features that are relatively similar to each other in the corresponding web page feature clusters; the representatives that are most similar to the web page features Determining a target set; updating the web page feature clusters associated with the determined representative set using the web page features; recomputing the representative set of updated web page feature clusters; and Includes extracting content from the web page based on the extraction template associated with the updated web page feature cluster.

According to another aspect of the invention, there is further provided an apparatus for extracting web page content, which comprises at least one processor, the processor configured to perform a method for extracting web page content. To be done.

According to another aspect of the present invention, there is further provided a computer-readable storage medium storing computer-readable program instructions, the program instructions implementing a method for extracting web page content when executed by a computer. You can

It is a figure which shows an example of the extraction waiting content in a web page. 6 is a flowchart of a web page content extraction method according to an exemplary embodiment of the present invention. It is a figure which shows a part of web page as an example. FIG. 3B is a diagram showing an example of converting a part of the web page shown in FIG. 3A into a DOM tree. 1 is a block diagram of a web page information extraction system according to an exemplary embodiment of the present invention. 1 is a configuration of a general-purpose device that can realize a web page content extraction device according to an exemplary embodiment of the present invention.

Hereinafter, preferred embodiments for carrying out the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that such an embodiment is merely an example and does not limit the present disclosure.

FIG. 1 is a diagram showing an example of contents waiting to be extracted on a web page.

Specifically, as an example, FIG. 1 shows a sample of one recruitment web page, of which the contents in a relatively small rectangular area on the left side and the contents in a relatively large rectangular area on the right side are awaiting extraction. It is a sample of the content. As shown in FIG. 1, the extraction waiting content in the relatively small rectangular area on the left side is information related to educational background, and the extraction waiting content in the relatively large rectangular area on the right side is information related to job content. In this web page sample, the contents waiting to be extracted relate to the information of educational background and job contents, but of course, the contents to be extracted depend on the needs of the user. Therefore, by using each embodiment of the present invention described below, Accordingly, necessary types of information can be extracted. Therefore, the content of the web page information waiting to be extracted may be text of a plurality of paragraphs or data existing in a table format. In addition, each embodiment of the present invention does not limit the type of the web page information content waiting to be extracted. The method and apparatus for extracting web page information according to each embodiment of the present invention can be applied to web pages of various formats and contents of various formats.

Hereinafter, the web page content extraction method according to the embodiment of the present invention will be described with reference to FIG. FIG. 2 is a flowchart of each step of the web page content extraction method according to the embodiment of the present invention.

Please refer to FIG. In step 201, the similarity between a web page feature and a representative set of at least one web page feature cluster is calculated.

The web page feature here may be, for example, a web page feature of a label of a document object model (DOM). According to each embodiment of the present invention, web pages can be obtained by scraping or downloading from multiple websites, and the scraped or downloaded web pages can be transformed into a document object model tree. FIG. 3B shows an example of the converted document object tree. After converting all scraped or downloaded web pages into a document object tree, the similarity between the web page features in each web page and a representative set of web page feature clusters is calculated in sequence. Alternatively, the processing of step 201 may be performed immediately after scraping one web page. Below is a brief introduction to the DOM tree model, a type of web page format.

FIG. 3A is a diagram showing a part of a web page as an example.

FIG. 3B is a diagram showing an example of converting a part of the web page shown in FIG. 3A into a document object model tree.

Please refer to FIG. 3B. Based on the definition of the leaf node pattern, as an example, the pattern of the leaf node “Kawasaki City” in FIG. 3B is “text_strong_p_div_Kawasaki City”, and the path of the leaf node pattern is “text_strong_p_div_”. ", and the content is "Kawasaki City". Similarly, the patterns of the other leaf nodes are “text_h2_div_Kawasaki City”, “text_p_div_Aso Ward Bento Office”, etc., respectively. “Text_strong_p_div_Kawasaki City” is a DOM label.

An example in which a DOM label is a web page feature will be described below. It should be noted that the example is for the purpose of illustration and does not limit the format of web page features. Embodiments of the present invention are not limited to web pages featuring DOM labels as web page features, but can be applied to any internet content. Here, the DOM is briefly explained. The Document Object Model DOM can access and modify the content and structure of a document in a platform and language independent manner. The DOM can dynamically change a user's page, for example by dynamically presenting or hiding one element, changing its attributes, incrementing one element, etc. to make the page interactive. Can be significantly improved. The DOM is actually a document model that is described in an object-oriented way, which defines the objects necessary to display and modify a document, the actions and attributes of these objects, and the relationships between these objects. it can. Although the DOM can be viewed as a tree-like representation of the data and structure on the page, the page can also be implemented in a non-DOM tree fashion, as shown in Figure 3B.

As an example, when the extraction-waiting web page content information is a DOM label, a sample of web page features as a DOM label can be expressed as follows.

“<tag1><tag2>……<tagn>C1”
One specific example in which a DOM label is a web page feature is as follows.

“<html><body><div[1]><div[2]><div[1]><div[4]><div[1]><div><dl><dt[6]text content ”
When the extraction-waiting web page content information is relational data and includes attributes and attribute values, one sample can be expressed as follows.

“<tag11><tag12>……<tag1n>R1<tag21><tag22>……<tag2n>C1”
Among them, R1 and C1 are an attribute and an attribute value that need to be extracted, respectively.

One specific example of web page features as relational data is as follows.

“<html><body><table><tbody><tr><td><table><tbody><tr[2]><td><table><tbody><tr[8]><td[1 ]>Required Education<html><body><table><tbody><tr><td><table><tbody><tr[2]><td><table><tbody><tr[8]><td[2]>4 Year Degree”
As can be seen from the above expression, “Required Education” is an attribute that represents “necessary educational background”, and “4 Year Degree” represents university graduation, that is, the attribute value of the attribute of “necessary educational background”. is there.

Thus, the web page feature sample is, for example, a DOM label-based sample or a relational data sample as described above. Of course, the web page characteristics are not limited to these two types of formats, and may be data in any appropriate format.

Given that there are k web page feature clusters waiting for extraction: {R1, R2,..., Rk}, and each cluster has its own few samples, k representative sets: {C1, C2 ,..., Ck}, where each representative set has some representative sample points, and the representative sets include web pages that have a relatively high degree of similarity between each other in the corresponding web page feature clusters. A sample of features is included.

As described above, after the web page of the extraction waiting web page content information is analyzed to obtain the sample of the web page feature, in step 201, the sample of the web page feature and the existing at least one web page feature cluster are included. Compute the degree of similarity of {R1, R2,..., Rk} to the representative set {C1, C2,..., Ck}. Note that at least one existing web page feature cluster {R1, R2,..., Rk} and its representative set {C1, C2,..., Ck} is a recently updated web page feature cluster and its representative set. Yes, since the initial seed web page feature cluster and its representative set are small, they may be determined by a person, and the By updating by iteration, the recently updated web page feature cluster {R1, R2,..., Rk} and its representative set {C1, C2,..., Ck} can be obtained. It should be noted that the web page feature sample here may be one or more samples, that is, a set of samples of web page information of extraction waiting information.

In step 201, the similarity between one sample of web page features and one representative set is calculated, and specifically, the sample of the web page features and each web page feature in the representative set are calculated. The similarity between the web page feature sample and the representative set can be calculated by calculating an average value of the calculated similarities. Note that the calculation of the degree of similarity according to the embodiment of the present invention is not limited to such a method, and any method that can calculate the degree of similarity between a sample and a set including other samples may be adopted. Hereinafter, a method of calculating the similarity between samples will be described.

<Calculation of similarity between samples>
Here, a method of calculating the similarity between samples will be described. Similarity calculation is used to calculate the similarity between samples. The sample here includes extraction waiting information and characteristics related to the information. In order to improve the accuracy of the similarity calculation, in particular, to improve the accuracy of the similarity calculation between web page formats of different representations, it is necessary to learn a method of calculating the similarity.

The task of similarity learning is to learn the similarity between samples. It is very difficult to calculate the similarity between samples due to the variety of web page expressions. The usual practice is to map the samples into another space in which the distance between like samples is close and the distance between like samples is long. Therefore, considering that the similarity between web page features is determined using a trained neural network, the learning of the similarity calculation in each embodiment of the present invention is realized by a network of two shared weights in parallel. , The network can be applied to classification problems that are highly categorized or where the entire training sample is not used in training the previous method. In the similarity calculation in each embodiment of the present invention, specifically, input samples to be paired (pair) are mapped to one space and parameters are adjusted so that the input sample pair Allow the distance in space to represent the categorization.

Then, FIG. 2 will be referred to again. In step 202, a representative set having the highest similarity to the sample of web page features is determined, and specifically, the similarity of the sample of web page features calculated in step 201 and each representative set is determined. By comparing the above, the representative set having the highest similarity with the sample of the web page feature can be determined.

After determining the representative set with the highest similarity to the sample of web page features, the web page features are used to update the web page feature clusters associated with the determined representative set in step 203 of FIG. Then, specifically, by merging the web page features into a determined representative set corresponding to the web page features, the web page feature clusters associated with the determined representative set are merged. To update.

Then, FIG. 2 will be referred to again. In step 204, the representative set of updated web page feature clusters is recomputed.

Let the number of representative sample points in the established representative set be M (M is an integer greater than 1). The number of representative sample points in the representative set may be a fixed value set in advance, or may be controlled using a predetermined threshold value, or another constraint. It may be one that can change according to conditions. For purposes of illustration, each embodiment of the present invention uses the following steps to update a representative sample point.

Using one sample in the set of web page feature samples in step 201, update the representative sample points for each cluster C _{m according} to the following formula:

By substituting formula (2) into formula (1) and performing the calculation, we select M samples before making the value of the formula on the right side of argmax of formula (1) the maximum (where , Argmax function is omitted), in which sim indicates the similarity, X _mi indicates the i-th sample in the classification C _m , and X _mj indicates the j-th sample in the classification m. Shown is the eye sample, where P _k is another cluster different from the cluster C _m to which X _mi belongs. Alternatively, according to the following formula (3), a sample whose similarity ratio is larger than a predetermined threshold value θ is selected.

After updating the web page feature cluster as described above, select another sample from the set of web page feature samples and set this step until convergence or a predetermined maximum number of iteration steps. Repeat until it reaches.

Finally, refer again to FIG. At step 205, content is extracted from the web page based on the extraction template associated with the updated web page feature cluster. Specifically, after obtaining each updated cluster, the extraction template associated with each updated cluster can be used to perform extraction on unknown data. The content in the data that matches the extraction template is extracted. In one example according to embodiments of the present invention, representative sample points are used to represent the extraction template. In such a case, when a new sample of awaiting extraction information is obtained, the similarity between the sample and a representative sample point of each updated cluster is calculated. Information extraction is performed on the sample by selecting the extraction template associated with the cluster having the highest degree of similarity and using the extraction template as the extraction template for extracting the sample. Note that it is well known to those skilled in the art to extract information using an extraction template associated with a certain cluster, so here, an application in an embodiment of the present invention will be described, but a specific implementation method will be described. Omit it.

FIG. 4 is a block diagram of the web page information extraction system in the embodiment of the present invention.

The system 400 shown in FIG. 4 includes a seed sample storage unit 401, a similarity learning unit 402, a similarity calculation unit 403, a representative point calculation unit 404, a representative point storage unit 405, a classification unit 406, an input unit 407 and an information extraction unit 408. including. Among them, the seed sample storage unit 401 stores the seed sample, the similarity learning unit 402 performs learning on the similarity between samples, the similarity calculation unit 403 calculates the similarity between samples, The representative point calculation unit 404 determines a representative point by calculation, the representative point storage unit 405 stores the representative point, and the classification unit 406 classifies the samples into corresponding clusters based on the similarity. Then, the input unit 407 inputs the web page, and the information extraction unit 408 extracts the web page content.

According to an embodiment of the present invention, a sample of web page features that are relatively similar to each other in a representative set is obtained from web page content of the same category. For example, the above-mentioned "required educational background" and "job content" belong to different categories. Further, the classification may be a classification indicating the type of web page feature. According to the embodiment of the present invention, the definition for the classification can be adaptively adjusted according to the needs of the application, and the adjustment for the definition of the classification is performed in the seed sample storage unit 401 shown in FIG. This is accomplished by manually adjusting to the memory seed sample. Furthermore, according to the embodiment of the present invention, the number of representative sets is equal to the number of types (types) of web page contents waiting to be extracted.

FIG. 5 is a configuration diagram of a general-purpose device 700 that can implement a web page information extraction device and a web page information extraction method according to an embodiment of the present invention. The general-purpose device 700 may be, for example, a computer system. It should be noted that the general-purpose device 700 is merely an example, and does not limit the use range or function of the method and apparatus according to the present invention. Also, the general purpose device 700 does not rely on any of the modules and assemblies in the methods and apparatus described above, or any combination thereof.

In FIG. 5, a central processing unit (CPU) 701 performs various processes based on a program stored in the ROM 702 or a program loaded from the storage unit 708 to the RAM 703. The RAM 703 can also store data necessary for the CPU 701 to perform various processes according to needs. The CPU 701, ROM 702 and RAM 703 are connected to each other via a buzz 704. Input/output interface 705 is also connected to bus 704.

Further, the input/output interface 705 is further connected with the following components, that is, an input unit 706 including a keyboard and the like, an output unit including a display such as a liquid crystal display (LCD) and a speaker, and the like. 707, a storage unit 708 including a hard disk, and a communication unit 709 including a network interface card such as a LAN card and a modem. The communication unit 709 performs communication processing via a network such as the Internet or LAN.

The drive 710 may be connected to the input/output interface 705 according to needs. The removable medium 711, such as a semiconductor memory, is set in the drive 710 as necessary, so that the computer program read from the medium can be installed in the storage unit 708.

The present invention also provides a program product including a machine-readable command code. Such instruction codes, when read and executed by a machine, can perform the methods in the embodiments of the present disclosure described above. Correspondingly, such program products are carried, for example, magnetic disks (including floppy disks (registered trademark)), optical disks (including CD-ROM and DVD), magneto-optical disks (MD (registered trademark)). Various storage media such as a semiconductor memory device are also included in the present disclosure.

The above-mentioned storage medium may include, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory device, but is not limited to these.

Further, each operation (process) in the above-described method can be realized by a method of a computer-executable program stored in various machine-readable storage media.

Further, the above embodiments and the like will be disclosed as additional notes as follows.

(Appendix 1)
A method for extracting web page contents,
Compute the similarity between a web page feature and a representative set of at least one web page feature cluster, said representative set being a web page with a relatively high degree of similarity between each other in the corresponding web page feature cluster. Includes a sample of features;
Determining a representative set with the highest similarity to the web page features;
Updating the web page feature cluster associated with the defined representative set using the web page features;
Recalculating a representative set of updated web page feature clusters; and extracting content from web pages based on an extraction template associated with the updated web page feature clusters.

(Appendix 2)
The method according to Appendix 1,
The method, wherein samples of web page features that are relatively similar to each other in the representative set are obtained from web page content of the same category.

(Appendix 3)
The method described in Appendix 2, of which
The method, wherein the number of the representative set is equal to the number (category) of classified web page contents.

(Appendix 4)
The method according to Appendix 2 or 3,
The method, wherein the categorization comprises a categorization representing a typology of web page features.

(Appendix 5)
The method according to Appendix 2 or 3,
The method wherein the adjustments to the classification definitions are achieved by adjusting the seed samples.

(Appendix 6)
The method according to Appendix 5,
The method wherein the seed sample is humanly determined.

(Appendix 7)
The method according to Appendix 1,
Based on the value (ratio) of the sum of the similarity between the web page features in the updated web page feature cluster and the other sample web page features divided by the sum of the similarity with each representative set, A method of selecting samples that make up a representative set of updated web page feature clusters.

(Appendix 8)
The method according to appendix 7,
A method of constructing a representative set of updated web page feature clusters using a predetermined number of samples corresponding to a relatively large proportion.

(Appendix 9)
The method according to Appendix 1,
A method of calculating similarity between web page features using a neural network.

(Appendix 10)
The method according to Appendix 9,
The method wherein the neural network comprises a Siamese network.

(Appendix 11)
The method according to Appendix 9,
The method, wherein the neural network is a trained neural network.

(Appendix 12)
The method according to Appendix 1,
A method in which the content information on the web page waiting to be extracted is a DOM label or relational data.

(Appendix 13)
A web page content extraction device,
Apparatus comprising at least one processor, which is configured to perform the method according to any one of notes 1-12.

(Appendix 14)
A computer-readable storage medium storing computer-readable program instructions, comprising:
A storage medium capable of implementing the method according to any one of appendices 1 to 12 when the program command is executed by a computer.

(Appendix 15)
A web page content extraction system,
A seed sample storage unit for storing seed samples;
A similarity learning unit for learning the similarity between samples;
A similarity calculation unit for calculating the similarity between samples;
A representative point calculation unit for determining a representative point by calculating;
Representative point storage unit for storing representative points;
A classification unit for classifying the samples into corresponding clusters based on the similarity;
A system comprising: an input unit for inputting a web page; and an information extracting unit for extracting the content of the web page.

Although the preferred embodiment of the present disclosure has been described above, the present disclosure is not limited to this embodiment, and all modifications to the present disclosure are within the technical scope of the present disclosure unless departing from the spirit of the present disclosure.

Claims (10)

A method of extracting web page contents,
Compute the similarity between a web page feature and a representative set of at least one web page feature cluster, said representative set being a web page with a relatively high degree of similarity between each other in the corresponding web page feature cluster. Includes a sample of features;
Determining a representative set with the highest similarity to the web page features;
Updating the web page feature clusters associated with the defined representative set using the web page features;
Recomputing a representative set of updated web page feature clusters; and extracting content from the web page based on an extraction template associated with the updated web page feature clusters. The method of claim 1, wherein
The method, wherein samples of web page features that are relatively similar to each other in the representative set are obtained from the same type of web page content. The method of claim 2, wherein
The method, wherein the number of said representative sets is equal to the number of awaiting web page contents. The method of claim 1, wherein
Based on the value (ratio) of the sum of the similarity between the web page features in the updated web page feature cluster and the other sample web page features divided by the sum of the similarity with each representative set, A method of selecting samples that make up a representative set of updated web page feature clusters. The method of claim 4, wherein
A method of constructing a representative set of updated web page feature clusters using a predetermined number of samples corresponding to a relatively large proportion. The method of claim 1, wherein
A method of determining similarity between web page features using a neural network. The method of claim 6, wherein
The method wherein the neural network comprises a Siamese network. The method of claim 6, wherein
The method, wherein the neural network is a trained neural network. A device for extracting web page contents,
Including at least one processor,
An apparatus, wherein the processor is configured to perform the method of any one of claims 1-8. A computer-readable storage medium that stores a program for causing a computer to execute the method according to any one of claims 1 to 8.

Images