CN101237465A - A Webpage Text Extraction Method Based on Fast Fourier Transform - Google Patents

Abstract

The invention discloses a web page text extraction method based on fast Fourier transform, comprising: reading in an HTML file, converting the file into a Unicode format, and storing it into a character array; performing window segmentation on the character array; The position in the document is statistically analyzed, and the intensity encoding conversion is performed on the characters according to the result to obtain the intensity value of the text, and each window character segment corresponds to an intensity value sequence; fast Fourier transform is performed on the intensity value sequence to obtain the F vector in the frequency domain; Calculate the distance between any two window character segments; set an interval for the window character segment, the interval is a combination of several consecutive windows, represented by a number pair (b, e), according to the distance between any two window character segments Calculate the weight of each interval; sort the weights of all intervals, and select the best text interval according to the weight. The invention has a high accuracy rate of extracting the text of the webpage, and can effectively distinguish the text from other parts of the webpage.

Description

A Webpage Text Extraction Method Based on Fast Fourier Transform

technical field

The invention relates to word information processing, in particular to a method for extracting webpage text based on fast Fourier transform.

Background technique

With the continuous development of the Internet and the substantial increase in the number of Web pages, the Web page has become a huge and widely distributed information source. A lot of information is included in the vast web, how to help people quickly extract effective information has become a very important issue.

According to the characteristics of HTML web pages, it is necessary to use the structural layout information of the web pages to segment the web pages, simulate the display mode of the IE browser, and analyze the web pages. According to the principle of human vision, the system divides the results of webpage parsing and processing into blocks, and then extracts the content of relevant webpage blocks required by users according to user needs. Therefore, webpage segmentation is a common method to extract effective information from webpages. Currently, the commonly used webpage segmentation methods mainly include the following:

1. Segmentation method based on positional relationship: This method uses the layout of the webpage to divide into blocks, divides a webpage into five parts: upper, lower, left, right and middle, and then classifies according to the characteristics of these five parts. However, the actual web page structure is much more complicated, and this method based on web page layout cannot be applied to all web pages; moreover, the granularity of web pages segmented by this method is relatively coarse, which may destroy the inherent characteristics of the web page itself, and it is difficult to fully include the entire web page. Semantic features of web pages.

2. Segmentation method based on Document Object Model (DOM, Document Object Model): This method finds the specific tags in the HTML document of the webpage, and uses the tag item to represent the HTML document as a DOM tree structure; and then includes the heading according to the specific tag , table, paragraph and list etc. to extract valid tree node data. But in many cases, the document object model is not used to represent the content structure of the webpage, so the semantic information of each block in the webpage cannot be distinguished accurately by using this method. Further descriptions of such methods can be found in Reference 1: "Wang Qi, Tang Shiwei, Yang Dongqing, Automatic Extraction of Webpage Topic Information Based on DOM [J], Computer Research and Development, 2004, 41(10): 1786-1791";

Reference 2: Hu Fei, Web Page Region Division and Search Method Based on Tag Tree [J], Computer Science, 2005, 32(8): 182-185.; Reference 3: Chang Yuhong, Jiang Zhe, Zhu Xiaoyan, Page Structure Analysis Based on Tag Tree Representation [J], Computer Engineering and Applications, 2004(16): 129-132.

Contents of the invention

The purpose of the present invention is to overcome the defect that the existing text extraction method cannot accurately define the text area and thus cannot accurately extract the text, thereby providing a text extraction method based on fast Fourier transform.

In order to achieve the above object, the present invention provides a method for extracting webpage text based on fast Fourier transform, which specifically includes the following steps:

Step 10), read in the HTML file, and convert the file into Unicode format, and store it in a character array;

Step 20), carry out window segmentation to the character array that step 10) obtains, the window character segment after segmentation comprises the character of fixed length;

Step 30), carry out statistical analysis to the position of character in document, according to the result of statistical analysis, character is carried out strength coding conversion, obtains the text strength value of this character, and each window character segment corresponds to a strength value sequence;

Step 40), carry out fast Fourier transform to the intensity value sequence of each window segment obtained in step 30), obtain the F vector of frequency domain;

Step 50), calculate the distance between any two window character segments according to the result of fast Fourier transform;

Step 60), interval is set for the window character segment, and described interval is the combination of several continuous windows, represents with numeral pair (b, e), according to the distance between any two window character segments obtained in step 50). Distance, calculate the weight of each interval;

Step 70), sort the weights of all intervals calculated in step 60), and select the best text interval according to the weights.

In the above technical solution, in the step 30), the results of the statistical analysis include the mean, standard deviation, and occurrence times of the characters in the document.

The calculation formula of the intensity value sequence is as follows:

I _{i, j} = M(W _{i, j} , i·l+j)=M(S _i·l+j , i·l+j), i=0Λ(w-1), j=0Λ(l- 1);

Among them, M is used to calculate the intensity value of a character, W represents the two-dimensional array of the window character segment, S represents the string array, i represents the number of the window character segment, j represents the position in the window character segment, and 1 represents the window character segment The length of , w represents the number of window character segments;

When calculating said M, for the character c appearing at position x, its text strength value is:

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; \&CenterDot;</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; c</span>}{<span\; class="notranslate">\; =</span>}^{<span\; class="notranslate">\; \&prime;</span>}{\mathrm{<span\; class="notranslate">\; a</span>}}^{<span\; class="notranslate">\; \&prime;</span>}{<span\; class="notranslate">\; ~</span>}^{<span\; class="notranslate">\; \&prime;</span>}{\mathrm{<span\; class="notranslate">\; z</span>}}^{<span\; class="notranslate">\; \&prime;</span>}{<span\; class="notranslate">\; ,</span>}^{<span\; class="notranslate">\; \&prime;</span>}{\mathrm{<span\; class="notranslate">\; A</span>}}^{<span\; class="notranslate">\; \&prime;</span>}{<span\; class="notranslate">\; ~</span>}^{<span\; class="notranslate">\; \&prime;</span>}{\mathrm{<span\; class="notranslate">\; Z</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; x</span>}\mathrm{<span\; class="notranslate">\; 0100</span>}<span\; class="notranslate">\; ~</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; xFFFF</span>}\\ {\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>{<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; otherwise</span>}\end{array}\right.$

In the above formula, μ _c is the mean value of the appearance position of the character c, σ _c is the standard deviation of the appearance position of the character c, and N _c is the number of occurrences of the character c.

In the above-mentioned technical solution, in the described step 50), the distance between any two sections of the calculation is to calculate the sum of the Euclidean distances on each frequency, and its calculation formula is as follows:

${\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; dis</span>}\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; ce</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\sqrt{{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

Wherein, F is the result after fast Fourier transform in step 40).

In the step 60), the weight of the calculation interval is the sum of the difference between groups minus the sum of the difference within the group, and the calculation formula of the weight of the interval is as follows:

V(b,e)=InterGroup(b,e)-IntraGroup(b,e)

$\mathrm{<span\; class="notranslate">\; InterGroup</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; Group</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; Group</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}$

$\mathrm{<span\; class="notranslate">\; IntraGroup</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; Group</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; Group</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}$

Wherein, IterGroup represents the difference between groups, IntraGroup represents the difference within a group, D _i,j represents the distance between any two window character segments calculated in step 50).

In the step 60), the calculation of the weight of each interval adopts the acceleration algorithm of the cumulative distance, and the calculation formula of the algorithm is as follows:

${\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;w</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;w</span>}$

$\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; a\&Lambda;b</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; c\&Lambda;d</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; +</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}}$

Among them, D _{x, y} represents the distance between the x segment and the y segment, D _{i, j} represents the 0, 1, ..., (i-1) window character segment and the 0, 1, ..., (j -1) The distance of window character segments.

In the above technical solution, in the step 70), the section with the largest weight is selected as the best text section.

In the above-mentioned technical solution, in the step 70), the calculation results of the step 60) are selected from the calculation results of the step 60) according to the order from large to small. The results of the weighted average select the best text interval.

The text information in the webpage is represented by a multi-byte character set, including Japanese, Korean and Chinese.

The advantages of the present invention are:

1. The present invention utilizes the frequency domain feature of the webpage to segment the webpage, filter the noise, and then extract effective information.

2. When the text content is long, the method of the present invention can effectively extract the text information of the webpage even if the page structure is complex and contains a variety of interference information, and distinguish the text from other parts of the page, and the extraction accuracy is high .

3. The present invention can extract the text content of the webpage without analyzing the specific webpage structure, has good versatility, and is applicable to webpages of different styles and themes.

Description of drawings

Fig. 1 is the flow chart of the webpage text extracting method based on fast Fourier transform of the present invention;

Fig. 2 a and Fig. 2 b are the schematic diagrams of the text strength function adopted when carrying out text strength coding in the present invention;

FIG. 3 is a schematic diagram of an accelerated algorithm for quickly calculating the sum of distances between consecutive intervals by using cumulative distances when calculating interval weights in the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

Before the method for extracting webpage text based on fast Fourier transform of the present invention is described, webpages are first classified according to the structural features of the webpage, specifically including the following categories:

Home page style - the home page of the website, generally contains multiple columns, pictures, animations, and links to several article titles. Such as: NetEase's home page.

Tabular style—information is given in the form of a list, generally listing several items in the form of a table, often with a pagination function. Example: A list of article titles for a forum forum.

Text style——refers to the bottom-level web page containing text content, generally only containing no more than one article content, with no or few comments. For example: the underlying web pages of various websites that contain a specific article.

Commentary style - in addition to the main text, there are several comments after the main text, represented by the forum.

The present invention is mainly aimed at the above-mentioned " text type " Chinese webpage to realize the extraction of webpage content. Text-style Chinese web pages usually contain a large section of text information, and some format information (such as navigation information, interactive information, JavaScript scripts, etc.) is placed before and after the text information.

Text information has the following characteristics:

1. Located in the middle of the HTML source file;

2. Mainly Chinese characters and English letters;

3. Relatively continuous text;

4. The signal characteristics of text information are similar;

5. The signal characteristics of text information and format information are different.

Format information has the following characteristics:

1. Located at the beginning and end of the HTML source file;

2. Mainly punctuation marks and English letters;

3. The signal characteristics of format information are similar;

4. The signal characteristics of format information and text information are different.

Analysis of the HTML document model shows that the document is a mixture of three types of signals, including:

1) HTML tag (TAG), in the form of "<tag><tag attribute=value></tag>".

For example:

<table width="756"border="0"align="center"></table>

2) Text natural language (TEXT), that is, sentences composed of Chinese and English characters. For example: about us Aboutus.

3) Script program (SCRIPT). For example: function MM_findObj(n,d){var p,i,x;if(!d)}

According to the structural features of text-style pages, the present invention converts the problem of text extraction into an HTML source file of a given bottom web page, and solves the optimal text interval. Below in conjunction with the example of a Chinese webpage, the specific implementation steps of the inventive method are described as follows:

Step 10, read in the HTML file, convert the file into Unicode format, and store it in a character array. The converted English letters are between 'a'～'z', 'A'～'Z', and the Chinese characters are between 0×0100～0×FFFF. The converted characters are stored in the character array S°, and the length of the character array is s°.

Suppose you read a webpage about Shangri-La, Yunnan on the NetEase Travel Channel, and after converting the webpage to Unicode format, the result of webpage conversion is as follows (due to the length of the original text, only part of the content is extracted in the following example):

“<!DOCTYPE html PUBLIC″-//W3C//DTD XHTML 1.0 Transitional//EN″

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<title>Cook Cantonese cuisine for Lama_Mango Netease Travel</title>

...

<! --page-->

<! --

<div class="tpage">

<span><a href="">Previous page</a></span><span class="fB"><a href="">1</a></span><span class=" fB″><a href="">2</a></span><span class="fBcDRed">3</span><span><a href="">Next</a>< /span>

</div>

-->

<div class="text" id="articlebody">

-------------------------------------------------- --------------

The above is the format information

-------------------------------------------------- --------------

The flow of people in the Old Town of Lijiang on July 14 is like the Shangxiajiu Pedestrian Street in Guangzhou, full of tourists, no exaggeration. Only in the stone street in the early morning can I see a small number of old Naxi people. Women of the Naxi ethnic group are more able to stick to their ancient beliefs, so they wear seven-star vests. Of course, you can also see the beautiful scenery in the morning when the professional and dedicated buddies bombard Lijiang with equipment with long guns and short cannons.

<br>&nbsp;&nbsp;&nbsp;&nbsp;Although the small bridge and flowing water are still there, the red lanterns at night are also very tempting, and they are all looking for a kind of excess passion. When I was walking down the street after listening to the charming Naxi ancient music alone, someone even saw me approaching me alone, enthusiastically introducing Mosuo girls to marry and dance, and asked if I Need to see a show, ? It was a surprise, as expected, openness brought everything. Of course, I don't believe that those girls are from the Mosuo ethnic group, and they just found other women to make up for it. Of course I didn't go. Later, when I was sitting casually in the bar, I heard from some donkey friends that there were even walking wedding groups in Guangdong just to experience walking marriages, which shows that people's pursuits are different.

<br>&nbsp;&nbsp;&nbsp;&nbsp;I remembered that there were green plums in Shangri-La in July, which were fresh. Friends who like sour know that green plums are ripe in June in the east. It should be the climate that delays its season. But the skin of Shangri-La’s greengage has turned yellow, but the flesh is not soft, it is still firm and even very hard. The sourness makes friends who like sour love it to death, and friends who are afraid of sour can make you sour to death. Not exaggerating at all. I bought it for 3 yuan a catty. You already know that I love sour food so much that I ate it for 3 days. Other lamas can't stand it after eating half of it, and the mouth membrane will peel off a layer of acid. But still miss that taste.

-------------------------------------------------- --------

The above is the text part

-------------------------------------------------- ------

<br></div>

<! --page-->

<! --

<div class="tpage">

<span><a href="">Previous page</a></span><span class="fB"><a href="">1</a></span><span class=" fB″><a href="">2</a></span><span class="fBcDRed">3</span><span><a href="">Next</a>< /span>

</div>

-->

...

//-->

</script>

<noscript>

<img src=″//secure-cn.imrworldwide.com/cgi-bin/m?ci=cn-netease&cg=0″alt=″″>

</noscript>

<! --END NNR Site Census V5.1-->

</body>

</html>

-------------------------------------------------- ----

The above is the format information

-------------------------------------------------- ----"

After the information of the above web page is converted into Unicode format, it is stored in a character array.

Step 20, perform window segmentation on the character array obtained in step 10. The window is used for sampling to select a segment of characters of equal length to implement Fourier transform in subsequent steps. Assuming that the size of the window is 1, the file contained in the character array S° is divided into several consecutive character segments with a length of 1, a total of w segments, and at the same time, the remaining characters less than 1 are deleted to obtain a new string array S, the length of the array is s. Use W to represent the two-dimensional array of the window, i to represent the window number, and j to represent the position in the window, then the calculation formula of the window is as follows:

${\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="0"\; label="S\; i"\; filenames="A20071006318200221.png,A20071006318200222.png"\; state="\{\{state\}\}">S</figure-callout></span><figure-callout\; id="0"\; label="S\; i"\; filenames="A20071006318200221.png,A20071006318200222.png"\; state="\{\{state\}\}">\; </figure-callout>}}_{\mathrm{<span\; class="notranslate">\; </span>}}^{\mathrm{<span\; class="notranslate">i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

W _i,j =S _i·l+j , i=0Λ(w-1), j=0Λ(l-1)

Still taking the above-mentioned webpage about Shangri-La as an example, window segmentation is performed on the character array storing the webpage. Assuming that the size of the window is set to 32, the result of the character array after window segmentation is as follows:

"Window[0]:??<!DOCTYPE html PUBLIC"-//W3C/

Window[1]: /DTD XHTML 1.0 Transitional//EN″

Window[2]: ? ? "http://www.w3.org/TR/xhtml1/

Window[3]: DTD/xhtml1-transitional.dtd″>??<

Window[4]: html xmlns="http://www.w3.org/19

Window[5]: 99/xhtml″>??<head>??<title>Promoting for Lamas

Window[6]: Dongcai_Mango Netease Travel</title>? ? <meta http-eq

...

Window[195]: span><span><a href="">next page</a></

Window[196]: span>? ? </div>? ? -->? ? <div class="

Window[197]: text″id=″articlebody″>7.14 Lijiang Old Town

Window[198]: The flow of people is like Shangxiajiu Pedestrian Street in Guangzhou, full of tourists, no exaggeration. stone only in the morning

Window[199]: I saw a few old Naxi people in Banjie.

Window[200]: star vest. Of course, you can also see the professional and dedicated buddies bombarding Li with equipment with long guns and short cannons.

Window[201]: The faint beauty of the river in the morning. ? ? <br>&nbsp;&nbsp;&nbsp

Window[202]:;&nbsp;Although the small bridge and flowing water are still there, the red lanterns at night are also very tempting, and they are all looking for

Window[203]: An excess of passion. When I was walking down the street after listening to the charming Naxi ancient music alone, I even

Window[204]: There is a very strange person who sees me coming up alone, and warmly introduces Mosuo girls walking marriage and dancing

Window[205]: Yanwu, do you want to watch the show? That was a surprise, openness brought everything, of course I didn't

Window[206]: I don’t believe those girls are from the Mosuo ethnic group, they just found women from other places to make up for it. of course i didn't go

Window[207]: Later, when I was sitting casually in a bar, I heard from some donkey friends that there were even walking wedding groups in Guangdong just for

Window[208]: For the experience of walking marriage, it shows that people have different pursuits. ? ? <br>&nbsp;&n

...

Window[599]: rite(_rsCL); ? ? //-->? ? </script>? ?

Window[600]: <noscript>? ? <img src="//secure-c

Window[601]: n.imrworldwide.com/cgi-bin/m? ci=

Window[602]: cn-netease & cg=0″alt=″″>??</

Window[603]: noscript>? ? <! --END NNR Site Cen

Window[604]: sus V5.1-->? ? </body>? ? </html>? ? "

It can be seen from the above example that the web page is divided into 605 sections.

Step 30, using the principle of statistics to convert the characters into intensity codes. In this step, for each character that appears in the document, analyze the rule that it appears in the entire document, and perform statistical analysis on the position of the character in the document to obtain the mean value, standard deviation, and The number of occurrences of this character in the document. Using the above mean, standard deviation, and number of occurrences, a text intensity value is calculated. When calculating the strength value of the text, the character segment divided in step 20 should be used as the calculation unit. For a character segment, code conversion is performed on the characters to obtain an intensity value sequence I, and the calculation formula of the intensity value sequence is as follows:

I _{i, j} = M(W _{i, j} , i·l+j)=M(S _i·l+j , i·l+j), i=0Λ(w-1), j=0Λ(l- 1)

Among them, M is used to calculate the intensity value of a character. For the character c appearing at position x, its text intensity value is:

In the above formula, μ _c is the mean value of the appearance position of the character c, σ _c is the standard deviation of the appearance position of the character c, and N _c is the number of occurrences of the character c.

Because the text contains more Chinese and English characters and less punctuation marks. Therefore, in the above formula, if the character is of text type (that is, 'a'～'z', 'A'～'Z', 0×0100～0×FFFF), the normal distribution formula is used as its encoding conversion function, The result is non-negative; for other characters, the formula of normal distribution plus offset is used as the conversion function, and the result is not positive. The above formula is a positive number for Chinese and English characters, and a negative number for punctuation marks. Because the text is located in the middle of the document, and the punctuation marks are located at both ends of the document, the formula concentrates the signal strength in the middle of the document for Chinese and English characters, and distributes the signal strength to both ends of the document for punctuation marks. The intensity function of the text is shown in Figure 2. It can be seen from the figure that since frequently occurring Chinese and English characters are often reflections of commonly used words, their signal strength increases proportionally; and frequently occurring punctuation marks are often typographical. Reflect, such as the less than and greater than signs, so their signal strength increases proportionally in the negative direction.

For all the character fields, after the above-mentioned similar operations, respective intensity value sequences can be obtained.

Step 40: Perform fast Fourier transform on the intensity value sequence of each window segment obtained in step 30 to obtain an F vector in the frequency domain. Its calculation formula is as follows:

F _i =FFT(I _i )

The specific implementation of the fast Fourier transform is a mature prior art, and will not be described in detail in this embodiment.

Step 50, calculate the distance between any two-character fields, the distance between the two-character fields is the sum of the Euclidean distances on each frequency. Its calculation formula is as follows:

${\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; dis</span>}\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; ce</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\sqrt{{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

It can be seen from the above formula that calculating the distance between any two segments actually calculates the difference between the two values at the corresponding frequency positions of the two segments, and then sums all the differences. For example, the distance between window A and window B is the difference between a0 and b0, ... the difference between a31 and b31, and the sum of the squares of these differences is then squared to obtain the sum of the Euclidean distances.

Step 60, setting intervals for the character segment, and calculating the weight of each interval. An interval is a combination of several consecutive windows, represented by a pair of numbers (b, e), which means that the interval represented by the pair of numbers is composed of windows W _b to W _e-1 , where 0≤ b<e≤w. After the interval is set, all the window segments in the file are divided into two groups, namely the interval internal group and the interval external group, the interval internal group A includes W _b ～W _e-1 , and the interval external group B includes W ₀ ～W _{b -1} and W _e ~W _w-1 . All window groups consist of the former part of group B {W ₀ , W ₁ , .., W _b-1 }, group A {W _b , W _b+1 , .., W _e-1 }, the rear part of group B A part of {W _e , W _e+1 , .., W _w-1 } is composed.

The weight of the interval refers to the sum of the difference between groups minus the sum of the difference within the group. The difference between groups refers to the difference between any segment in group A inside the interval and any segment in group B outside the interval. The sum of is the inter-group difference; the intra-group difference refers to the difference between the inner group A and the outer group B for any two segments within the interval, and the sum of the differences is the intra-group difference. The formula for calculating interval weights is as follows:

V(b,e)=InterGroup(b,e)-IntraGroup(b,e)

$\mathrm{<span\; class="notranslate">\; InterGroup</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; Group</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; Group</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}$

$\mathrm{<span\; class="notranslate">\; IntraGroup</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; Group</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; Group</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}$

In this step, a preferred implementation manner of calculating the weight of the interval is to use an accelerated algorithm for accumulating distance, and the sum of differences between two consecutive groups can be quickly calculated by using this algorithm. As shown in Figure 3, the calculation formula of the algorithm is as follows:

${\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;j</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;w</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; \&Lambda;w</span>}$

$\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; a\&Lambda;b</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; c\&Lambda;d</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; +</span>{\stackrel{<span\; class="notranslate">\; \&OverBar;</span>}{\mathrm{<span\; class="notranslate">\; D.</span>}}}_{\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}}$

Among them, D _{x, y} represents the distance between the x segment and the y segment, D _{i, j} represents the 0, 1, ..., (i-1) window character segment and the 0, 1, ..., (j -1) The distance of window character segments. The above formula is used to speed up the calculation of inter-group difference and intra-group difference. First calculate the cumulative value table, and then quickly calculate the inter-group difference and intra-group difference by looking up the table and simple algebraic operations. Among them, D _{i, j} , i=1Λw, j=1Λw is the accumulated value table.

Step 70, sort the weights of all intervals calculated in step 60, and the interval with the largest weight is the best text interval. In step 60, since the set interval contains all possible situations of continuous window combinations, weights of multiple intervals will be obtained in the end, these weights are sorted in descending order, and finally the weights are selected The largest interval is used as the best text interval, and the content contained in the best text interval is the text to be finally extracted from the webpage in the present invention.

For the aforementioned webpage about Shangri-La, select the interval with the largest weight. According to the result of the weight calculation in step 60, the maximum weight is 1.8671557984059033E9. The b of the interval with the largest weight is 197, and e is 395. This interval is the Find the best text interval.

In an embodiment, another implementation manner of sorting the interval weights and selecting the best text interval is to perform weighted average on the weights, and then obtain the best text interval in the average sense according to the result of the weighted average. During implementation, the weighted average is usually performed on the intervals whose weights are greater than 0, and the optimal text interval (b ^* , e ^* ) in the average sense is calculated. The formula for calculating the weighted average is as follows:

$<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span>}{\underset{\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span>}$

Among them, V(b, e) represents the interval weight.

Still taking the aforementioned webpage about Shangri-La as an example, from the weight calculation results in step 60, assuming that there are 100 intervals with weights greater than 0, these weights and corresponding intervals are as follows:

No.1: Area{b=197e=395w=1.8671557984059033E9}

No.2: Area{b=198e=395w=1.865928902944519E9}

No.3: Area{b=197e=394w=1.863446434026815E9}

No.4: Area{b=198e=394w=1.8620946999597936E9}

No.5: Area{b=197e=396w=1.8534012640629482E9}

No.6: Area{b=196e=395w=1.8533969765727189E9}

No.7: Area{b=198e=396w=1.852261927708008E9}

No.8: Area{b＝199e＝395w＝1.8511999688045855E9}

No.9: Area{b=197e=393w=1.8500594430878716E9}

No.10: Area{b=196e=394w=1.849788102344682E9}

No.11: Area{b=198e=393w=1.848510799009436E9}

No.12: Area{b=199e=394w=1.8471652124879038E9}

No.13: Area{b＝197e＝397w＝1.8453086053177962E9}

No.14: Area{b=195e=395w=1.845281305908179E9}

No.15: Area{b＝198e＝397w＝1.8442583536949947E9}

No.16: Area{b=195e=394w=1.8417764283302329E9}

No.17: Area{b=197e=392w=1.8413777475416255E9}

No.18: Area{b＝198e＝392w＝1.8396801709467006E9}

No.19: Area{b＝196e＝396w＝1.8396421919565065E9}

No.20: Area{b＝200e＝395w＝1.838057893744711E9}

No.21: Area{b＝199e＝396w＝1.8377040837184753E9}

No.22: Area{b=196e=393w=1.8365645973901665E9}

No.23: Area{b＝200e＝394w＝1.8338399474557528E9}

No.24: Area{b＝199e＝393w＝1.8333431983722968E9}

No.25: Area{b＝201e＝395w＝1.832882136920093E9}

No.26: Area{b＝194e＝395w＝1.8327158264980187E9}

No.27: Area{b＝197e＝398w＝1.8317380757017245E9}

No.28: Area{b=196e=397w=1.8315166911690896E9}

No.29: Area{b＝195e＝396w＝1.8314938196044166E9}

No.30: Area{b＝198e＝398w＝1.8307755060003867E9}

No.31: Area{b＝202e＝395w＝1.830544198380903E9}

No.32: Area{b＝199e＝397w＝1.829861304277684E9}

No.33: Area{b＝194e＝394w＝1.829311678505044E9}

No.34: Area{b＝195e＝393w＝1.828719245958915E9}

No.35: Area{b＝201e＝394w＝1.8285160965672174E9}

No.36: Area{b＝196e＝392w＝1.828012158947821E9}

No.37: Area{b＝197e＝391w＝1.8270460014801817E9}

No.38: Area{b＝202e＝394w＝1.8260582076294603E9}

No.39: Area{b＝198e＝391w＝1.8251564260435276E9}

No.40: Area{b＝200e＝396w＝1.824723891564548E9}

No.41: Area{b=199e=392w=1.8243151092166026E9}

No.42: Area{b=195e=397w=1.8233264390587733E9}

No.43: Area{b＝203e＝395w＝1.822325780416904E9}

No.44: Area{b＝195e＝392w＝1.8202939671958587E9}

No.45: Area{b＝200e＝393w＝1.8198227669199252E9}

No.46: Area{b＝201e＝396w＝1.8196575937589269E9}

No.47: Area{b＝193e＝395w＝1.8191558800920327E9}

No.48: Area{b＝194e＝396w＝1.8189200336928308E9}

No.49: Area{b＝197e＝399w＝1.8179459850346885E9}

No.50: Area{b＝196e＝398w＝1.8179439755481179E9}

No.51: Area{b＝203e＝394w＝1.8176838100122943E9}

No.52: Area{b=202e=396w=1.8174102756958842E9}

No.53: Area{b＝198e＝399w＝1.817070992891399E9}

No.54: Area{b＝200e＝397w＝1.8170506741580334E9}

No.55: Area{b＝199e＝398w＝1.8165496617398362E9}

No.56: Area{b＝194e＝393w＝1.8164182449130914E9}

No.57: Area{b＝193e＝394w＝1.8158518234459796E9}

No.58: Area{b＝201e＝393w＝1.8143022038707862E9}

No.59: Area{b＝196e＝391w＝1.8138511011079237E9}

No.60: Area{b＝197e＝390w＝1.813416825235355E9}

No.61: Area{b=201e=397w=1.812101903347275E9}

No.62: Area{b＝202e＝393w＝1.8116598519465666E9}

No.63: Area{b＝198e＝390w＝1.8113552225214372E9}

No.64: Area{b＝194e＝397w＝1.810719247254324E9}

No.65: Area{b＝200e＝392w＝1.8106092331069574E9}

No.66: Area{b＝202e＝397w＝1.8099494719208207E9}

No.67: Area{b＝195e＝398w＝1.809720873865331E9}

No.68: Area{b＝199e＝391w＝1.8095815493579323E9}

No.69: Area{b=203e=396w=1.8093194340361586E9}

No.70: Area{b＝204e＝395w＝1.8091673410619712E9}

No.71: Area{b＝194e＝392w＝1.8081203284794781E9}

No.72: Area{b＝195e＝391w＝1.8062889464577138E9}

No.73: Area{b＝192e＝395w＝1.8055887898178735E9}

No.74: Area{b＝193e＝396w＝1.8053577759523911E9}

No.75: Area{b＝201e＝392w＝1.8049212023955352E9}

No.76: Area{b＝197e＝400w＝1.804362583413403E9}

No.77: Area{b=204e=394w=1.8043406024255657E9}

No.78: Area{b＝196e＝399w＝1.8041515829944117E9}

No.79: Area{b＝200e＝398w＝1.8039011525318637E9}

No.80: Area{b＝198e＝400w＝1.8035751666398578E9}

No.81: Area{b＝193e＝393w＝1.80312176475147E9}

No.82: Area{b＝203e＝393w＝1.8030793314788742E9}

No.83: Area{b＝199e＝399w＝1.8030163410122762E9}

No.84: Area{b＝192e＝394w＝1.8023851986898751E9}

No.85: Area{b＝202e＝392w＝1.8021209078151228E9}

No.86: Area{b＝203e＝397w＝1.8019899976293116E9}

No.87: Area{b＝196e＝390w＝1.8003818327393115E9}

No.88: Area{b＝201e＝398w＝1.799061835030309E9}

No.89: Area{b＝191e＝395w＝1.797390318129374E9}

No.90: Area{b＝193e＝397w＝1.7971241276820748E9}

No.91: Area{b＝194e＝398w＝1.797104678286477E9}

No.92: Area{b＝202e＝398w＝1.797000014978798E9}

No.93: Area{b＝204e＝396w＝1.796316784871037E9}

No.94: Area{b＝195e＝399w＝1.7958957929261835E9}

No.95: Area{b＝200e＝391w＝1.7956939769691014E9}

No.96: Area{b＝199e＝390w＝1.7955746426529288E9}

No.97: Area{b＝205e＝395w＝1.7951057911539783E9}

No.98: Area{b＝193e＝392w＝1.7949530569627554E9}

No.99: Area{b＝194e＝391w＝1.7942824448319867E9}

No.100: Area{b=191e=394w=1.79426301425113E9}

According to the aforementioned calculation formula, the weighted average of the above weights results in begin=182.3652086633145, end=404.76999807248177, and the corresponding optimal text interval can be obtained according to the weighted average.

Adopt method of the present invention, can obtain good actual effect:

In one instance, randomly select NetEase Travel ( http://ok.travel.163.com/itinerar/list.jsp ), e-Travel World ( http://www.eyooworld.com/index.html ), Hongxiu Tim Hongxiu ( http://www.hongxiu.com/ ), Shuimu Forum ( www.newsmth.net ), and Keyuan Xingkong Forum ( www.kyxk.net ) are used for the experiment. Select 50 pages each, for a total of 250 pages.

Manually observe the start and end positions of the text in the source code, that is, the correct text interval, denoted as (B, E); the interval with the largest weight given by the program running results, that is, the optimal text interval, denoted as (b1, e1 ); the interval obtained by weighted average, that is, the best text interval in the average sense, is denoted as (b ^* , e ^* ). The total number of paragraphs of the HTML source code after processing is denoted as w, then the weight method is used to obtain the optimal text interval accuracy R, and the weighted average method is used to obtain the optimal text interval accuracy R ^* .

$\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\frac{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; b</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; +</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; w</span>}}<span\; class="notranslate">\; ,</span>$ ${\mathrm{<span\; class="notranslate">\; R</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\frac{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; +</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; w</span>}}$

Table 1 below shows the accuracy results of extracting text intervals from the above web pages.

NetEase Travel e travel the world Red sleeves add fragrance Mizuki Forum Keyuan Star Forum R mean 0.988153706 0.913867141 0.985827381 0.968767584 0.972534604 R ^* mean 0.944079847 0.882985277 0.91369307 0.958057645 0.929748895

Table 1

From the experimental results, it can be seen that the accuracy of the algorithm for text content extraction of web pages with different structures is high. The mean R values are all above 90%, and the mean R values of the five websites are about 96.583%. There are four types of websites whose R ^* averages are above 90%, and the averages of five websites are about 91.957%.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than limit them. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent replacements to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all of them should be included in the scope of the present invention. within the scope of the claims.

Claims (9)

1. webpage context extraction method based on fast fourier transform specifically comprises following steps:

Step 10), read in html file, and this document is converted to Unicode format, and deposit in the character array;

Step 20), character array that step 10) is obtained carries out windowed segments, the window character field after the segmentation comprises the character of regular length;

Step 30), statistical analysis is carried out in the position of character in document, character is carried out the intensity coding conversion, obtain the text intensity level of this character, the corresponding intensity level sequence of each window character field according to The result of statistics;

Step 40), to step 30) in the intensity level sequence of each window character field of obtaining carry out fast fourier transform, obtain the F vector of frequency domain;

Step 50), calculate distance between any two window character fields according to the result of fast fourier transform;

Step 60), be between window character field setting district, described interval is the combination of several continuous windows, to the distance between any two window character fields of obtaining in (b, e) expression is according to step 50), calculates each interval weights with numeral;

Step 70), to step 60) in calculate resulting all interval weights orderings, select between best text area according to weights.

2. the webpage context extraction method based on fast fourier transform according to claim 1, it is characterized in that, in described step 30) in, described The result of statistics comprises average, the standard variance that occurs the position about character, and the occurrence number of character in document.

3. the webpage context extraction method based on fast fourier transform according to claim 2 is characterized in that, the computing formula of described intensity level sequence is as follows:

I _i，j＝M(W _i，j，i·l+j)＝M(S _i·l+j，i·l+j)，i＝0Λ(w-1)，j＝0Λ(l-1)；

Wherein, M is used to calculate the intensity level of a character, and W represents the two-dimensional array of window character field, S represents the character string array, and i represents the numbering of window character field, and j represents the position in the window character field, l represents the length of window character field, and w represents the number of window character field;

When calculating described M, for the character c that occurs at position x, its text intensity level is:

In the above-mentioned formula, μ _cBe the average that the position appears in character c, σ _cBe the standard variance that the position appears in character c, N _cIt is the number of times that character c occurs.

4. the webpage context extraction method based on fast fourier transform according to claim 1, it is characterized in that, in described step 50) in, the distance between any two sections of the described calculating is for calculating the summation of the Euclidean distance on each frequency, and its computing formula is as follows:

$D_{i,j}=\mathrm{dis}\tan \mathrm{ce}(F_i,F_j)=\underset{k=0\mathrm{\&Lambda;}(l-1)}{\mathrm{\&Sigma;}}\sqrt{{\left|\right|F_{i,k}-F_{j,k}\left|\right|}^2}$

Wherein, F is a step 40) in be result after the fast fourier transform.

5. the webpage context extraction method based on fast fourier transform according to claim 4, it is characterized in that, in described step 60) in, the weights of described computation interval are that difference sum between group is deducted group interpolation sum, the computing formula of described interval right weight is as follows:

V(b，e)＝InterGroup(b，e)-IntraGroup(b，e)

$\mathrm{InterGroup}(b,e)=\underset{\mathrm{Group}\left(i\right)\&NotEqual;\mathrm{Group}\left(j\right)}{\mathrm{\&Sigma;}}{D}_{i,j}$

$\mathrm{IntraGroup}(b,e)=\underset{\mathrm{Group}\left(i\right)\&NotEqual;\mathrm{Group}\left(j\right)}{\mathrm{\&Sigma;}}{D}_{i,j}$

Wherein, it is poor that InterGroup represents between group, and IntraGroup represents to organize interpolation, D _{I, j}Expression step 50) distance between any two the window character fields that calculate in.

6. the webpage context extraction method based on fast fourier transform according to claim 5 is characterized in that, in described step 60) in, each interval weights of described calculating adopt the accelerating algorithm of cumulative distance, and the computing formula of described algorithm is as follows:

${\stackrel{\&OverBar;}{D}}_{i,j}=\underset{x=0\mathrm{\&Lambda;i}-1,y=0\mathrm{\&Lambda;j}-1}{\mathrm{\&Sigma;}}{D}_{x,y},i=1\mathrm{\&Lambda;w},j=1\mathrm{\&Lambda;w}$

$\underset{i=\mathrm{a\&Lambda;b}-1,j=\mathrm{c\&Lambda;d}-1}{\mathrm{\&Sigma;}}{D}_{i,j}={\stackrel{\&OverBar;}{D}}_{b,d}-{\stackrel{\&OverBar;}{D}}_{a,d}-{\stackrel{\&OverBar;}{D}}_{b,c}+{\stackrel{\&OverBar;}{D}}_{a,c}$

Wherein, D _{X, y}The distance of expression x section and y section, D _{I, j}Represent the 0th, 1 ..., (i-1) individual window character field and the 0th, 1 ..., the distance of (j-1) individual window character field.

7. the webpage context extraction method based on fast fourier transform according to claim 1 is characterized in that, in described step 70) in, selecting the interval of weights maximum is between best text area.

8. the webpage context extraction method based on fast fourier transform according to claim 1, it is characterized in that, in described step 70) in, from step 60) result of calculation according to from big to small selective sequential weights greater than 0 interval, these interval pairing weights are done weighted average, select between best text area according to average weighted result.

9. the webpage context extraction method based on fast fourier transform according to claim 1 is characterized in that, the multibyte character set representations of the text message in the described webpage comprises Japanese, Korean and Chinese.

Images