CN108897749A - Method for abstracting web page information and system based on syntax tree and text block density - Google Patents

Abstract

The invention relates to a web page information extraction method based on syntax tree and text block density, comprising: acquiring title text information of web page; setting screening threshold, calculating text block densities of all nodes of the web page, if the text block density is greater than the screening The threshold node is the collection node, and the node text information of the collection node is extracted; if the number of the collection node is 1, the text information of the node is used as the target information for extraction; if the number of the collection node is greater than 1, the title The text information and the node text information are respectively converted into the title deep syntax tree and the node deep syntax tree that uniquely express the semantics of the sentence; the overall similarity between each node deep syntax tree and the title deep syntax tree is obtained, and the overall similarity The node text information corresponding to the maximum value of is extracted as the target information.

Description

Web page information extraction method and system based on syntax tree and text block density

technical field

The invention belongs to the field of network data collection, in particular to a method and system for extracting web page information based on syntax tree semantic recognition and tag text block density ratio.

Background technique

With the rapid development of information technology, data is becoming more and more electronic, and fast and effective network data collection technology has become particularly important. Effective network data collection is an inevitable requirement for enterprises to analyze the market environment and customer needs. Enterprises with efficient data collection capabilities show strong competitiveness in the era of big data. At the same time, efficient data collection technology is also related to national political security. Today, with the maturity of information technology, various forms of information are disseminated rapidly on the Internet, the body of netizens is becoming larger and larger, public opinion is changing rapidly, and the speed of dissemination of ideas and speech is like a wild horse, which also poses new challenges for the ability to control public opinion. In this case, the efficient collection of network information has become particularly important, as it provides necessary network data for the network research room, which is related to the political security of the country.

Now due to technical needs, network data collection technology has blossomed everywhere in recent years, and collection technologies for different web page data emerge in endlessly. One of the more difficult technical difficulties is the extraction of short text information on web pages. When the body text of the webpage is short, it becomes more difficult to identify the main body information, because compared with webpages with long body text, and useless information such as advertisements in the webpage, the recognition degree of "noise" is lower. When filtering webpage information, it is more likely to mistakenly filter and exclude it as spam information, and instead extract some advertising information by mistake as the main body of the text.

Contents of the invention

In view of the above problems, the present invention proposes a method for extracting web page information based on syntax tree and text block density, including: obtaining the title text information of the web page through a regular expression; setting a screening threshold to calculate the text block density of all nodes of the web page, Take the node whose text block density is greater than the screening threshold as the collection node, and extract the node text information of the collection node; if the number of collection nodes is 1, then use the text information of this node as the target information to extract; if the total number of collection nodes is greater than 1, The text information of the title and the text information of the node are converted into a title syntax tree and a node syntax tree respectively through a probabilistic context-free model; the title syntax tree and the node syntax tree are respectively converted into a title that uniquely expresses the semantics of a sentence through a synchronous tree replacement grammar Deep syntax tree and node deep syntax tree; calculate the overall similarity between the title deep syntax tree and each node deep syntax tree, and extract the node text information corresponding to the maximum value in the overall similarity as the target information.

In the web page information extraction method of the present invention, the text block density is obtained in the following manner:

Among them, TBD(v) is the text block density of node v, v.children is the child node set of node v, v _i is the child node of node v, CN _vi is the number of text characters contained in the text block of child node v _i , LCN _vi is the number of hyperlink characters contained in the text block of sub-node v _i , TN _vi is the number of labels contained in the text block of sub-node v _i , LTN _vi is the number of tags contained in the text block of sub-node v _i The number of hyperlink tags.

The method for extracting web page information of the present invention, when the total number of collection nodes is greater than 1, carry out the following steps:

Extract the title word vector t _i of the title deep grammar tree, and the text word vector a _i of the node deep grammar tree with the same structure as the title deep grammar tree; use the title word vector t _i and the text word vector a _i word vector similarity The overall similarity S=S ₁ ·S ₂ ·S ₃ ·...·S _{n is} obtained; wherein, 0<i≤n, n is a positive integer, and n is the number of deep syntax tree nodes of the title.

The present invention also relates to a web page information extraction system based on syntax tree and text block density, comprising:

The text information acquisition module is used to obtain the title text information of the webpage through regular expressions, and the node text information of the collection node; including setting the screening threshold, calculating the text block density of all nodes on the web page, and taking the text block density greater than the screening threshold The node is a collection node, and the node text information of the collection node is extracted;

The first target information acquisition module is used to acquire target information when the number of node text information is 1;

The second target information acquisition module is used to obtain target information for extraction when the number of node text information is greater than 1; wherein the title text information and the node text information are respectively converted into title syntax trees and node grammars through a probabilistic context-free model tree; the title syntax tree and the node syntax tree are respectively converted into a title deep syntax tree and a node deep syntax tree that uniquely express sentence semantics by synchronizing the tree replacement grammar; obtaining the overall similarity between the title deep syntax tree and each node deep syntax tree, Extract the node text information corresponding to the maximum value in the overall similarity.

In the web page information extraction system of the present invention, in the text information acquisition module, the text block density is obtained in the following manner:

Among them, TBD(v) is the text block density of node v, v.children is the child node set of node v, v _i is the child node of node v, CN _vi is the number of text characters contained in the text block of child node v _i , LCN _vi is the number of hyperlink characters contained in the text block of sub-node v _i , TN _vi is the number of labels contained in the text block of sub-node v _i , LTN _vi is the number of tags contained in the text block of sub-node v _i The number of hyperlink tags.

In the web page information extraction system of the present invention, the second target information acquisition module specifically includes:

The word vector acquisition module is used to obtain the title word vector t of the title deep grammar tree, and the text word vector a of the node deep grammar tree having the same structure as the title deep grammar tree;

The similarity acquisition module is used to obtain the word vector similarity of the title word vector t _i and the text word vector a _i The overall similarity S=S ₁ ·S ₂ ·S ₃ ····S _{n is} obtained; where 0<i≤n, n is a positive integer, and n is the number of deep syntax tree nodes of the title.

Description of drawings

FIG. 1 is a flowchart of a method for extracting web page information according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, a method and system for extracting webpage information based on syntax tree and text block density proposed by the present invention will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described here are only used to explain the present invention, and are not intended to limit the present invention.

The purpose of the present invention is to solve the deficiencies of the prior art, and proposes a data collection method for processing web pages with texts of different lengths. By adjusting the result selection of the traditional data collection algorithm based on the tag-to-text density ratio, the selection of one target tag is changed to the selection of multiple target tags with similar tag densities, so that when the length of the text body is short , the situation that the main body of the text is filtered out by mistake will no longer happen; news headlines that are easy to obtain, have high collection accuracy, and can generally summarize the semantics of the main body of the text will be used as the comparison object for the semantics of the main body of the text, so that the method of matching the body of the web page through semantics has semantics Compare the standard; analyze the node syntax tree of multiple candidate texts and article titles, construct the node syntax tree, and make preliminary preparations for semantic matching later; perform syntax tree deformation processing on all the constructed syntax trees, and convert the syntax tree Extract the principal components in the text, retain the key words and structures such as subject-predicate-object, and unify the sentences with different sentence structures but the same semantics in form, so as to prepare for the comparison of text semantics in the future; for the pre-processed Multiple text body syntax trees and news headline syntax trees perform overall semantic matching of the syntax trees, and identify which short text is the collection object through semantics.

FIG. 1 is a flowchart of a method for extracting web page information according to an embodiment of the present invention. As shown in Figure 1, the steps of the web page information extraction method of the present invention are as follows:

Step S1, obtaining the title text information of the webpage;

Step S2, running the web page label text density algorithm;

Step S3, setting a screening threshold, screening collection nodes, and extracting node text information of collection nodes;

Step S4, judging the number of collection nodes;

Step S5, if there is only one collection node, the node text information of the collection node is used as the target information to extract it;

Step S6, if the number of collected nodes is greater than 1, then process the title text information and the node text information respectively to generate a title syntax tree and a node syntax tree;

Step S7, performing normalization processing on all syntax trees to generate title deep syntax trees and node deep syntax trees respectively;

Step S8, calculating the overall similarity between the title deep syntax tree and the node deep syntax tree;

Step S9, select the node text information corresponding to the maximum value of the overall similarity as the target information, and extract it.

Specifically, in the embodiment of the present invention, the aforementioned data is matched through a regular expression dedicated to identifying the title and author of the article, the time of publication, and other fixed-position, uniform-form data to obtain information such as the title of the article.

Next, calculate the text block density TBD of each node according to calculation formula (1);

Among them, TBD (textblockdensity) is the text block density; let v be a node in the web page parsing tree T, Blk (v) be the text block with node v as the root node, define the text block density TBD (v) of node v as The sum of the ratio of the number of non-link text characters to the number of non-link labels in all text blocks rooted at the child nodes of node v.

CN (ContentNumber) is the number of characters in the text block, that is, the number of text characters contained in the text block; usually, the text under the body text block is relatively concentrated, and the length of the text characters will be relatively large; the text under the noise text block is scattered, and the text The character length will be smaller.

LCN (LinkContentNumber) is the number of hyperlink characters in the text block, that is, the number of hyperlink characters contained in the text block; the hyperlink text under the text text block is relatively small, and the hyperlink text under the noise text block is relatively large.

TN (Tag Number) is the number of tags in the text block, that is, the number of tags contained in the text block; the body text block is mostly continuous text, and the number of tags is small; the noise text block is scattered text, and the number of tags is large.

LTN (LinkTagNumber) is the number of hyperlink tags in the text block, that is, the number of hyperlink tags contained in the text block; the text under the hyperlink tags is mostly noise information, and the number of hyperlink tags contained in the body text block is small, noise The number of hyperlink tags contained under the text block is large.

After obtaining the text block density TBD of all nodes, a screening threshold is set, and all nodes whose TBD value is greater than the screening threshold are designated as collection nodes, and the text information in all above collection nodes is extracted.

If the number of collection nodes obtained through the screening threshold is 1, that is, the obtained text information is unique, then the node text information of this collection node is extracted as the target information; if the number of collection nodes obtained through the screening threshold is greater than 1, that is, the obtained The text information of the title text information is not unique, it is necessary to process the aforementioned title text information and the node text information of the collection node to obtain the node text information with the highest similarity to the title text information as the target information for extraction.

In the embodiment of the present invention, the probabilistic context-free model (PCFG) is used to preprocess the title text information and the node text information, that is, the PCFG model is used to analyze and generate the title syntax tree and the node syntax tree of the title text information and the node text information respectively . PCFG model is a commonly used natural language syntax analysis model. The analysis algorithm of PCFG is the same as that of non-probabilistic context-free grammar. They all start to expand from non-terminal symbols, and calculate the corresponding probability for each different analysis tree through PCFG. When the sentence has ambiguity, the probability is calculated to select which grammatical analysis result, and the selection criterion is the highest generation probability. Let T be an alternative tree. When the sentence is ambiguous, the analysis result T* of the sentence can be selected through probability, namely:

The generation probability of the analysis candidate tree T is the product of the conditional probabilities of all the rules required to generate T:

Where r is the rule, and P(r) is the probability of satisfying this rule.

As a mature natural language analysis model, PCFG has a certain ability to disambiguate and generate syntax trees with high precision. Moreover, due to the Markov nature of the model itself, it does not consider the context environment, so it is not sensitive to the sparsity of the data, so its analysis results have certain robustness.

A further step is to process all syntax trees (title syntax trees and node syntax trees) generated above. The embodiment of the present invention uses Synchronous Tree Substitution Grammar (STSG) to convert all title syntax trees and node syntax trees into title deep syntax trees and node deep syntax trees respectively.

The transformation grammar mentioned here is a theory aimed at the relationship between sentence syntax and the inner semantics of sentences. This theory believes that all natural language sentences have two structures: deep and surface; the surface structure is the human eye recorded in the document. The visible text is the actual text sequence; the deep structure of a sentence is different from the superficial structure of a sentence, and the deep structure of a sentence actually determines the actual semantics of a sentence; multiple sentences with the same semantics but different superficial structures correspond to the same A deep sentence structure.

Example: My lunch today is a hamburger.

I had a hamburger for lunch today.

Although the structures of these two sentences are different, the inner deep sentence structure is exactly the same, because they express the same meaning.

STSG is a rule self-learning algorithm based on grammar trees, which learns grammar rules by itself through grammar trees. Convert the surface structure of a sentence into a deep structure, so that sentences with different syntax but the same semantics can generate the same sentence syntax tree.

The basic rule extraction algorithm of STSG is as follows:

Input: syntax tree pair <T(f), T(e), A>, A is the alignment relationship between T(f) and T(e).

Create an empty base rule set P.

t(p) is a subtree of T(f) whose root node is p;

t(q) is a subtree of T(e) whose root node is q;

A(t(p),t(q)) is the word alignment relationship related to t(p) and t(q) in A

If<t(p),t(q),A(t(p),t(q))>satisfies word alignment restrictions and syntactic restrictions

Then add <t(p), t(q), A(t(p), t(q))> to the rule set P

Output: basic rule set P

Use the trained STSG algorithm to standardize all sentences corresponding to the title syntax tree and the node syntax tree, and convert the title syntax tree and the node syntax tree into a title deep syntax tree and a node deep syntax tree that uniquely express the sentence semantics. For the multiple syntax trees that may be generated by the text in the same tag, standardize them one by one and attribute them to this tag.

Word vector is a language model for natural language processing. Its core idea is to map different words or words in the language into a high-dimensional vector through different semantic standards. Each dimension of these vectors is composed of real numbers. The relationship between words concretizes the abstract semantic relationship between words, so that the computer can approximate the abstract semantic relationship through specific calculations. The directional similarity between word vectors also reflects the semantic similarity between words.

After obtaining the title deep grammar tree and the node deep grammar tree, it is necessary to compare the semantic similarity between the grammar trees through the trained word vector. Matches all heading deep syntax trees with node deep syntax trees from within the same text.

The matching process adopts the preorder traversal method of the tree. Here T is the title deep syntax tree, trees A ₁ , A ₂ , A ₃ ,..., A _m are m syntax trees from the text deep syntax tree L _i to be matched with tree T; here m is the number of candidate texts, m and i are positive integers, 0<i≤m; match the m trees sequentially. T and A _i perform preorder traversal synchronously. If T and A _i have the same structure, start to calculate the similarity of the two trees. If not, skip A _i . When calculating the similarity, for the node a in the same position ∈T, b∈A _i , let its word vectors be t _i and a _i , calculate the cosine similarity between t _i and a _i to get the word vector similarity S _i . If T has n nodes in total, 0<i≤n, the overall similarity between tree A _i and tree T Finally, the A _i corresponding to the maximum similarity value of T is taken as the collection target, and the corresponding node text information is taken as the target information for extraction.

The specific algorithm is as follows:

The input is a set of trees corresponding to the title tree T and the text L _i {A ₁ ,A ₂ ,A ₃ ,…,A _n }

1,

2. Perform synchronous preorder traversal on A _i and T, t _i and a _i are the word vectors corresponding to the traversed nodes, S _i is the semantic similarity between T and L _i , S=1;

3. If both t _i and a _i are not empty, calculate At the same time S=S·S _i , otherwise skip A _i and go to step 1;

4. Find the S corresponding to each tree A _i above;

The output is the overall _similarity between T and Li.

After the overall similarity calculation is completed, the node text information corresponding to the maximum value in the overall similarity between the node text information and the title text information is selected as the final acquisition target text for extraction.

The present invention uses titles that are easy to collect through template matching, have high collection accuracy, and can highly summarize the semantics of the text body as the semantic matching standard, perform semantic matching processing on multiple short texts that are suspected of being text bodies, and use the text with the highest degree of semantic matching as the semantic matching standard. Collection objects. In the case that it is difficult to accurately screen the objects to be collected through information such as webpage tags, the semantics of multiple short texts that are suspected of being the main text of the webpage are used as screening information to provide a semantics-based webpage data collection method, which greatly breaks through the previously impossible Limitations of collection methods that use the collection object's own semantics for recognition.

Claims (10)

1. A webpage information extraction method based on syntax tree and text block density, is characterized in that, comprises: Obtain the title text information of the webpage; set the screening threshold, calculate the text block density of all nodes of the web page, and use the node whose text block density is greater than the screening threshold as the collection node to extract the node text information of the collection node; If the number of the collection node is 1, the text information of the node is used as the target information for extraction; If the number of the collected nodes is greater than 1, the title text information and the node text information are respectively converted into a title deep syntax tree and a node deep syntax tree uniquely expressing sentence semantics; each node deep syntax tree and the title deep syntax tree are obtained For the overall similarity of the syntax tree, the node text information corresponding to the maximum value in the overall similarity is used as the target information for extraction. 2. The method for extracting web page information according to claim 1, characterized in that the title text information is obtained through a regular expression. 3. The method for extracting web page information as claimed in claim 1, wherein the text block density is obtained in the following manner: Among them, TBD(v) is the text block density of the node v, v.children is the child node set of the node v, v _i is the child node of the node v, CN _vi is the text block contained in the child node v _i LCN _vi is the number of hyperlink characters contained in the text block of the sub-node v _i , TN _vi is the number of labels contained in the text block of the sub-node v _i , LTN _vi is the number of tags contained in the sub-node v i The number of hyperlink tags contained in the text block of _i . 4. The method for extracting web page information as claimed in claim 1, wherein the title text information and the node text information are converted into a title syntax tree and a node syntax tree respectively by using a probabilistic context-free model, and a synchronous tree is used to replace The grammar converts the heading syntax tree and the node-deep syntax tree into the heading-deep syntax tree and the node-deep syntax tree, respectively. 5. The method for extracting webpage information as claimed in claim 1, wherein the overall similarity is obtained by the following steps: Extracting the title word vector t _i of the deep syntax tree of the title, and the text word vector a _i of the deep syntax tree of the node having the same structure as the title deep syntax tree; Take the word vector similarity between the title word vector t _i and the text word vector a _i Obtain the overall similarity S=S ₁ ·S ₂ ·S ₃ ·····S _n ; Where 0<i≤n, n is a positive integer, and n is the number of deep syntax tree nodes of the title. 6. A web page information extraction system based on syntax tree and text block density, characterized in that it comprises: The text information acquisition module is used to obtain the title text information of the webpage, and collect the node text information of the nodes; it includes setting a screening threshold, calculating the text block density of all nodes in the web page, and taking the text block density greater than the screening threshold. The node is the collection node, and the node text information of the collection node is extracted; The first target information acquisition module is used to acquire target information for extraction when the number of text information of the node is 1; The second target information acquisition module is used to obtain target information for extraction when the number of the node text information is greater than 1; wherein the title text information and the node text information are respectively converted into a title deep syntax tree and a node that uniquely express the semantics of the sentence Deep syntax tree: Obtain the overall similarity between the deep syntax tree of each node and the deep syntax tree of the title, and use the node text information corresponding to the maximum value in the overall similarity as the target information. 7. The web page information extraction system according to claim 6, wherein, in the text information acquisition module, the title text information is acquired through a regular expression. 8. The web page information extraction system according to claim 6, wherein, in the text information acquisition module, the text block density is obtained in the following manner: Among them, TBD(v) is the text block density of the node v, v.children is the child node set of the node v, v _i is the child node of the node v, CN _vi is the text block contained in the child node v _i LCN _vi is the number of hyperlink characters contained in the text block of the sub-node v _i , TN _vi is the number of labels contained in the text block of the sub-node v _i , LTN _vi is the number of tags contained in the sub-node v i The number of hyperlink tags contained in the text block of _i . 9. The webpage information extraction system as claimed in claim 6, wherein, in the second target information acquisition module, a probabilistic context-free model is used to convert the title text information and the node text information into a title syntax tree respectively and the node syntax tree, and convert the heading syntax tree and the node syntax tree into the heading deep syntax tree and the node deep syntax tree respectively by using a synchronous tree replacement grammar. 10. The web page information extraction system according to claim 6, wherein the second target information acquisition module further comprises: The word vector obtaining module is used to obtain the title word vector t _i of the deep syntax tree of the title, and the text word vector a _i of the deep syntax tree of the node with the same structure as the title deep syntax tree; The similarity acquisition module is used to obtain the word vector similarity of the title word vector t _i and the text word vector a _i The overall similarity S=S ₁ ·S ₂ ·S ₃ ····S _{n is} obtained; where 0<i≤n, n is a positive integer, and n is the number of deep syntax tree nodes of the title.