CN103853834B - Text structure analysis-based Web document abstract generation method - Google Patents

Abstract

The invention discloses a text structure analysis-based Web document abstract generation method. The method comprises the steps of using a URL (uniform resource locator) as input, integrating the webpage main bodies of visual features and text features for extraction, partitioning the main bodies into a plurality of semantic paragraphs, and abstracting each semantic paragraph, so the generated abstract has higher coverage rate. The text structure analysis-based Web document summary generation method realizes the generation of the text abstract with better quality from a Webpage aiming at the conditions that the Webpage structure is complex, the main body is hard to identify and the Chinese automatic abstract is still positioned in the probe stage.

Description

A Web Document Summary Generation Method Based on Text Structure Analysis

technical field

The invention relates to the technical fields of web page text extraction, natural language processing, and Chinese automatic summarization, in particular to a generation method of web document summaries based on text structure analysis.

Background technique

At present, the Internet has become the main source for people to obtain information. Especially with the rapid development of User Generated Content (UGC) in recent years, the information on the Internet is growing explosively. Although search engines can return search results according to user requirements. But users still need to find the most suitable webpage for their needs from the search list, especially because there are a large number of search engine optimization and reposting phenomena on the Internet, which brings great difficulties to users to find information quickly and accurately.

The automatic summarization system uses computers to quickly process web documents, and captures the core content of web documents according to a certain compression ratio, from which users can obtain subject information and judge the value of the web documents, which improves the efficiency of users searching for information.

There is a large amount of noise information in Web documents, such as advertisements, navigation bars, user function bars, related recommendations, copyright information and other information that has nothing to do with the topic. Web document is a kind of semi-structured information. Although it has a certain structure, its semantics cannot be determined. The representation of the content in the HTML source code and the final rendered page can be very different. In recent years, the extensive application of JS and AJAX technology has made web page data no longer static HTML codes, but dynamically generated, and even corresponding changes will be made to the user's operation behavior. So how to extract the content related to the theme and with the correct structure from the Web document, there is a certain difficulty.

The research on Chinese automatic summarization system has a history of more than 20 years, but it is still in the exploratory stage, and the results of automatic summarization are far from satisfactory. The methods of automatic summarization are mainly divided into two categories, automatic summarization based on understanding and automatic summarization based on extraction. Since there is still no major breakthrough in natural language processing technology, the method based on comprehension cannot truly realize automatic summarization.

The research history of automatic summarization technology for Web documents is even shorter. "Compared with traditional texts, the text structure of web pages is loose, the title naming is relatively less rigorous, and there may be no terminator at the end of a sentence, and there are a large number of inconsistencies with the text." Relevant content, which brings certain difficulties to the generation of summaries.”

Contents of the invention

The purpose of the present invention is to provide a method for generating Web document summaries based on text structure analysis. This method comprehensively uses technologies such as visual feature analysis, natural language analysis, and text structure analysis to generate semantically based summaries for each web page in the search results. A good, high-quality webpage summary to provide users with reference.

The purpose of the present invention is achieved like this:

A method for generating web document summaries based on text structure analysis, which includes the following steps:

1) Input the URL of the webpage to be summarized;

2) Extracting the webpage text from the webpage to be summarized based on visual analysis, specifically including;

2.1) The browser core is used to parse and render the Web document;

2.2) Use the visual tree (VIPS) algorithm to divide the webpage into blocks to obtain the position and area of each block;

2.3) Carry out word segmentation to each block;

2.4) Analyze text features for each block;

2.5) Scoring whether each block contains text;

2.6) Concatenate texts with scores higher than a certain threshold in order;

2.7) output the text of the Web document;

3) Carry out automatic summarization based on text structure analysis to the extracted text, specifically including:

3.1) obtain the webpage text by step 2);

3.2) Carry out word segmentation and part-of-speech tagging on the text;

3.3) Perform text preprocessing: identify the basic structure in the text, that is, identify the title of the article, and complete sentence and paragraph segmentation;

3.4) Segment the text into semantic segments, and identify the position where semantic conversion occurs through text structure analysis, as a symbol of semantic segment segmentation;

3.5) For each semantic segment, use the generalization method of TFIDF to measure the importance of each sentence in the semantic segment, and then extract several sentences that can best represent the theme of the semantic segment according to the word count requirements of the abstract;

3.6) Connect the sentences in order to output the abstract.

The text features in the step 2.4) are word count, font size, declarative sentence quantity, non-declarative sentence quantity and text fragment quantity.

Described in the step 2.5) to judge whether each block contains text to score, use the following formula to calculate the score of scoring:

$\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}$

Among them, S represents the number of declarative sentences, N represents the number of non-declarative sentences, P is a value calculated according to the size and position of the block, x ₁ , y ₁ represent the coordinates of the upper left corner of the block, x ₂ , y ₂ represent the coordinates of the lower right corner of the block coordinate.

The analysis and identification of the position where semantic conversion occurs in the step 3.4) is:

3.4-1) segment the document D into sentences, and each two adjacent sentences are undetermined segmentation points;

3.4-2) Score each undetermined split point, the formula is:

$\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; a</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; i</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>$

Among them, R(s _i , s _j ) represents the inter-sentence semantic correlation between sentence s _i and sentence s _j ; p _i represents that the segmentation point is between sentence s _i and s _i-1 , if Q(p _i )＞Q (p _i-1 ) and Q(p _i )>Q(p _i+1 ), it means that p _i is the maximum value point of the segmentation point weight, so p _i is the segmentation point between semantic segments in the text. a is an adjustable empirical parameter, indicating the range of semantic analysis when identifying the segmentation point, that is, considering the a sentences before and after the segmentation point.

3.4-3) If the score of the segmentation point is greater than a certain threshold and is a local maximum, that is, the score is higher than the scores of the two previous and subsequent segmentation points, the segmentation point is the segmentation point of the semantic segment, that is, step 3.4) Where the semantic translation occurs as described in .

The analysis and recognition step 2) of the position where the semantic conversion occurs The calculation of the inter-sentence semantic correlation comprises the following steps:

3.4-2-1) Segment the sentence into a collection of words;

3.4-2-2) Use the following formula to calculate the semantic correlation between sentences

$\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>$

Among them, R(w _i , w _j ) represents the inter-word semantic correlation between word w _i and word w _j .

In the step 3.5), the importance of each sentence in the semantic segment is measured using the following formula:

V(S ₁ )=sum(w∈S ₁ )*TFIDF(w)

Among them, when calculating TFIDF(w), each paragraph is regarded as an independent file, and several paragraphs included in the whole article are regarded as a file set.

The invention can filter out the words and links irrelevant to the theme in the webpage, and identify the main text of the article contained in the webpage, with high accuracy and high robustness. The automatic summarization process adopts the automatic summarization technology based on text structure analysis, and the generated summaries have high coverage and smooth summaries.

According to the compression ratio specified by the user, the present invention can form a relatively accurate and smooth summary that can cover the meaning of the original text within a few seconds only by inputting the URL address of the webpage to be summarized, helping users to quickly and accurately looking for information on the Internet.

Description of drawings

Fig. 1 is a flowchart of the present invention;

Fig. 2 is the flow chart of webpage preprocessing of the present invention;

Fig. 3 is a flow chart of the automatic summarization of the present invention.

detailed description

The invention discloses a search engine-oriented method for generating web document abstracts, which can automatically analyze a web page and generate text abstracts reflecting the theme of the web page.

The invention includes a webpage body text extraction that combines visual features and text features and an automatic text summary based on subtopic division through text structure analysis.

The invention takes a URL as input, and finally generates a text summary through two stages of web page text extraction and automatic summary.

The following is a further description of the specific algorithms of the two stages, combined with an example of summarizing a news web page:

Figure 1 describes the overall process from the URL to be summarized to generating the summary, which includes the webpage preprocessing process and the automatic summarization process.

Specifically, in the embodiment, the present invention acquires the URL of the news webpage to be summarized in the URL input step of the webpage preprocessing flow (see FIG. 2 ). The web page preprocessing process can more accurately find the body part of the web page by analyzing the visual features, which is more robust than other methods. At the same time, text features, text correlation analysis, HTML tag features, semantic features and other features are considered comprehensively to further improve the accuracy of Web page text extraction.

The webpage rendering step is responsible for reading the webpage corresponding to the input URL. In this embodiment, the IE11 browser core is used to process the HTML tags and render the webpage. On the basis of webpage rendering, the visual tree analysis step adopts the VIPS algorithm to analyze the visual tree of the webpage to obtain the position and area of each block. In this embodiment, this step divides the news web page to be summarized into 6 blocks: a top block, a bottom block, a navigation block, an advertisement block and two blocks containing text. The word segmentation step is responsible for word segmentation of each block. Then, the text feature analysis step performs text feature analysis on the word segmentation result. Finally, in the comprehensive analysis step, the features and text features of each block obtained by the visual tree analysis are comprehensively analyzed, and the news text is output.

In this example, P(x ₁ , y ₁ , x ₂ , y ₂ ) was calculated using the following formula.

P(x ₁ , y ₁ , x ₂ , y ₂ )=(x ₂ -x ₁ )*(y ₂ -y ₁ )-x ₁ *y ₁

Among them, x ₁ and y ₁ represent the coordinates of the upper left corner of the block, and x ₂ and y ₂ represent the coordinates of the lower right corner of the block. Then calculate the V(s) value of each block:

$\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}$

The V(s) values of the above six blocks from large to small are 3.7×10 ⁶ , 2.3×10 ⁶ , 7.5×10 ⁵ , 5.4×10 ⁶ , 3.7×10 ⁵ , 1.6×10 ⁵ , 1.2×10 ⁴ .

In this embodiment, the threshold value used is 10 ⁶ , so the blocks with V(s) greater than 10 ⁶ are selected, that is, the two blocks with the largest V(s) values. In this embodiment, the two blocks with the largest V(s) values are the two blocks containing the text, so the news text is correctly extracted.

After the news text is extracted, the automatic summarization process (see Figure 3) is followed, including the steps of text preprocessing, inter-word correlation calculation, inter-sentence correlation calculation, semantic segment segmentation, and summary generation.

A text preprocessing step to identify the basic structure in the text, that is, identify the title of the article, and complete sentence and paragraph segmentation. In this embodiment, the news text contains 8 paragraphs and 23 sentences.

The inter-word correlation calculation step is based on the computational semantics knowledge provided by HowNet, and the correlation between two words is obtained by calculating the sememe similarity of the two words. The formula used is as follows:

R(w ₁ , w ₂ )=max(Rele(C _i , C _j ))(C _i ∈ w ₁ , C _j ∈ w ₂ )

Among them, R(w ₁ , w ₂ ) represents the semantic correlation between two words, Rele(C _i , C _j ) represents the correlation between two sememes, and the maximum value represents the semantic correlation between two words .

The inter-sentence correlation step obtains the correlation between the two sentences by analyzing the correlation between words in the two sentences.

$\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>$

Among them, R(s ₁ , s ₂ ) represents the correlation between two sentences, for each word in sentence 1, find the word with the highest correlation in sentence 2, and calculate the correlation between these two words Spend. Finally, these maximum values are summed to obtain the correlation between the two sentences.

A semantic segment segmentation step refers to the document "Research on Text Structure Analysis Method Based on Content Correlation Calculation" to analyze text structure. The feature of the segmentation point between semantic segments is that the first sentence after the segmentation point has a small correlation with the previous sentences, but has a large correlation with the subsequent sentences. 22 segmentation points between 23 sentences in this embodiment are adopted to calculate the score of segmentation points, and find the maximum point of function Q (pi):

$\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; a</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; i</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>$

In this embodiment, Q(p _i ) contains two maximum points, and the news is divided into three semantic segments according to these two maximum points. Each semantic segment contains a subtopic of the news. In this embodiment, the first semantic segment is an overview of the news event, and the last two semantic segments are comments on the news event by two parties.

A summary generation step, according to user requirements, extracts a summary from the main body in text format in a certain proportion.

In this embodiment, the summary generation step calculates the sum of the correlations between sentences in each subtopic and article title vocabulary sequences through the inter-sentence correlation calculation step, so as to determine the value of each subtopic. The number of sentences extracted from a subtopic is proportional to the relevance of the subtopic to the article title.

In this embodiment, the ratio specified by the user is 0.2, that is, 5 sentences out of 23 sentences are extracted to form a summary. By calculating the value of the 3 subtopics, it is determined to extract 2, 1, and 1 sentences from the 3 semantic segments respectively. Finally, the summary generation step connects the selected 5 summary sentences in order to form a summary and output it.

Claims (5)

1. A method for generating web document summaries based on text structure analysis, characterized in that: the method may further comprise the steps: 1) Input the URL of the webpage to be summarized; 2) Extracting the webpage text from the webpage to be summarized based on visual analysis, specifically including; 2.1) The browser core is used to parse and render the Web document; 2.2) Use the visual tree algorithm to divide the webpage into blocks to obtain the position and area of each block; 2.3) Carry out word segmentation to each block; 2.4) Analyze text features for each block; 2.5) Score whether each block contains text, and use the following formula to calculate the scoring value: $\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}$ Among them, S represents the number of declarative sentences, N represents the number of non-declarative sentences, P is a value calculated according to the size and position of the block, x ₁ and y ₁ represent the coordinates of the upper left corner of the block, x ₂ and y ₂ represent the coordinates of the lower right corner of the block coordinate; 2.6) Concatenate texts with scores higher than a certain threshold in sequence; 2.7) output the text of the Web document; 3) Carry out automatic summarization based on text structure analysis to the extracted text, specifically including: 3.1) obtain the webpage text by step 2); 3.2) Carry out word segmentation and part-of-speech tagging on the text; 3.3) Perform text preprocessing: identify the basic structure in the text, that is, identify the title of the article, and complete sentence and paragraph segmentation; 3.4) Segment the text into semantic segments, and identify the position where semantic conversion occurs through text structure analysis, as a symbol of semantic segment segmentation; 3.5) For each semantic segment, use the generalization method of TFIDF to measure the importance of each sentence in the semantic segment, and then extract several sentences that can best represent the theme of the semantic segment according to the number of words in the abstract; 3.6) Connect the sentences in order to output the abstract. 2. The method according to claim 1, characterized in that: the text features described in step 2.4) are word count, font size, declarative sentence quantity, non-declarative sentence quantity and text fragment quantity. 3. The method according to claim 1, characterized in that: in step 3.4), the analysis and recognition of the position where semantic conversion occurs is: 3.4-1) segment the document D into sentences, and each two adjacent sentences are undetermined segmentation points; 3.4-2) Score each undetermined split point, the formula is: $\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; a</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>$ Among them, R(s _i , s _j ) represents the inter-sentence semantic correlation between sentence s _i and sentence s _j ; p _i represents that the segmentation point is between sentence s _i and s _i-1 , if Q(p _i )＞Q (p _i-1 ) and Q(p _i )>Q(p _i+1 ), indicating that p _i is the maximum value point of the segmentation point weight, so p _i is the segmentation point between semantic segments in the text; a is an adjustable empirical parameter, representing the scope of semantic analysis when identifying the segmentation point, that is, considering each a sentence before and after the segmentation point; 3.4-3) If the score of the segmentation point is greater than a certain threshold and is a local maximum, that is, the score is higher than the scores of the two previous and subsequent segmentation points, the segmentation point is the segmentation point of the semantic segment, that is, step 3.4) Where the semantic translation occurs as described in . 4. method according to claim 3, it is characterized in that: step 3.4-2) the calculation of the inter-sentence semantic correlation degree comprises the following steps: 3.4-2-1) Segment the sentence into a collection of words; 3.4-2-2) Use the following formula to calculate the semantic correlation between sentences $\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>$ Among them, R(w _i , w _j ) represents the inter-word semantic correlation between word w _i and word w _j . 5. method according to claim 1, it is characterized in that: described in step 3.5) measure the importance of each sentence in place semantic segment and use following formula to calculate: V(S ₁ )=sum(w∈S ₁ )*TFIDF(w) Among them, when calculating TFIDF(w), each paragraph is regarded as an independent file, and several paragraphs included in the whole article are regarded as a file set.