TWI611308B - Webpage data extraction device and webpage data extraction method thereof - Google Patents

Abstract

A webpage data extraction device and a webpage data acquisition method thereof. Web page data extraction device execution: according to the address relevance of the URL of the webpage data, the webpage data is divided into URL groups; the first webpage data and the second webpage data are selected from the webpage data of the URL group; the first webpage is parsed The data and the second webpage data are obtained by the webpage node data collection; according to the path relevance of the XML path language of the webpage node data of the webpage node data collection and the textual relevance of the text content, the webpage node data of the webpage node data collection is divided into plural webpages a node data group; respectively calculating a sum of text content of each webpage node data group; determining a main webpage node data group of the webpage node data group according to the sum of the text content; according to the webpage node data included in the main webpage node data group The XML path language determines the main content of the web page to capture information.

Description

Web data capture device and webpage data acquisition method thereof

The present invention relates to a webpage data extraction device and a webpage data extraction method thereof. More specifically, the present invention relates to an automated webpage data retrieval device and a webpage data acquisition method thereof.

With the development of Internet applications, all kinds of information can be obtained from different web pages. Therefore, when there is a specific data analysis requirement, the main content of the webpage of the relevant website can be extracted and analyzed.

However, in the main content extraction methods of the webpage, the manual crawling analysis is used. However, the main content judgment of different webpages of different websites is artificially performed, and the efficiency is rather unsatisfactory. Accordingly, in order to improve the efficiency of the main content of the webpage, there are custom-made programs, and various templates and layouts of the webpage are used as training data for webpage analysis and main content. Take the technology.

However, the way of this kind of custom program can only be processed for the template and layout of a specific web page. Therefore, when the webpage revision or its grammatical structure is slightly adjusted, if the corresponding adjustment is not made for the custom program, it will lead to analysis and Take the result Obvious error.

Moreover, due to the increasingly complex layout of web pages, the amount of web page information has also increased substantially. The number of webpage nodes of a single web page may be as high as nearly one thousand. According to this, when the structure or type of webpage changes, the customization is made. The complexity of the program adjustment will be more difficult, and may even require rewriting the custom program. As a result, the main content of the web page is judged to be inefficient.

Therefore, how to improve the shortcomings of the main content of the traditional webpage to achieve inefficiency is the goal of the industry to work together.

The main object of the present invention is to provide a webpage data extraction method for a webpage data retrieval device. The webpage data retrieval device receives the plurality of webpage data from the webpage server. The webpage data extraction method comprises: (a) causing the webpage data retrieval device to divide the plurality of webpage data into at least one URL group according to the address association of a plurality of uniform resource locators (URLs) of the plurality of webpage data. . The at least one URL group includes a first URL group, the first URL group includes at least a portion of the plurality of webpage materials, and (b) the webpage data retrieval device selects a portion of the plurality of webpage materials from the first URL group. a web page data and a second web page data; (c) a web page data capture device that parses the first web page data and the second web page data to obtain a web page node data set. The webpage node data set includes a plurality of webpage node materials, and each webpage node data includes a corresponding XML Path Language and text content.

The aforementioned method for extracting webpage data further includes: (d) making webpage materials The capturing device divides the plurality of webpage node data of the webpage node data set into a plurality of webpage node data groups according to the path association of the plurality of XML path languages of the plurality of webpage node data of the webpage node data collection and the textual relevance of the plurality of textual content. The webpage data group includes at least part of the plurality of webpage node data; (e) causing the webpage data extracting device to calculate the sum of the textual contents of the plurality of webpage node data of each webpage node data group; (f) making the webpage data The capturing device determines at least one primary webpage node data group of the plurality of webpage node data groups according to the sum of the plurality of textual contents; (g) causing the webpage data extracting apparatus to include at least a plurality of webpage nodes included in the at least one main webpage node data group The plural XML path language of the data determines the main content of the web page to capture information.

To achieve the above objective, the present invention discloses a webpage data capture device, including a receiving unit and a processing unit. The receiving unit is configured to receive a plurality of webpage materials from the web server. The processing unit is configured to: divide the plurality of webpage materials into at least one URL group according to address relevance of the plurality of URLs of the plurality of webpage materials. The at least one URL group includes a first URL group, and the first URL group includes at least part of the plurality of webpage materials; and the first webpage data and the second webpage data are selected from the plurality of webpage materials of the first URL group; Parsing the first webpage data and the second webpage data into a webpage node data collection. The webpage node data set includes a plurality of webpage node materials, and each webpage node data includes a corresponding XML path language and text content.

The processing unit is further configured to: according to the path association of the plurality of XML path languages of the plurality of webpage node data of the webpage node data set and the textual relevance of the plurality of textual content, the plurality of webpage nodes of the webpage node data collection It is divided into multiple web node data groups. Wherein, each webpage node data group includes at least part of the plurality of webpage node data; respectively, calculating a sum of text content of a plurality of webpage node data of each webpage node data group; and judging at least a plurality of webpage node datagroups according to the sum of the plural textual content sums A main webpage node data group; determining a main content of the webpage to retrieve information according to a plurality of XML path languages of at least one of the plurality of webpage node data groups.

Further objects of the present invention, as well as the technical means and implementations of the present invention, will become apparent to those skilled in the art in view of the appended claims.

1, 2‧‧‧ web data capture device

11, 21‧‧‧ receiving unit

13, 23‧‧ ‧ processing unit

Wp‧‧‧Webpage

Ul‧‧‧Uniform Resource Addresser

Ug‧‧‧At least one URL group

UL1‧‧‧First URL Group

WP1‧‧‧First webpage information

WP2‧‧‧Second webpage information

ND‧‧‧Website Node Information

NDX‧‧‧XML path language

NDT‧‧‧ text content

Wpg‧‧‧Website node data collection

Ndg‧‧‧Website Node Data Group

MNDG‧‧‧ at least one main web node data group

The main content of the MX‧‧‧ webpage captures information

1A is a block diagram of a webpage data capture operation according to a first embodiment of the present invention; FIG. 1B is a block diagram of a webpage data capture apparatus according to a first embodiment of the present invention; and FIG. 2A is a webpage of a second embodiment of the present invention; FIG. 2B is a block diagram of a webpage data capture device according to a second embodiment of the present invention; and FIG. 3 is a flowchart of a webpage data capture method according to a third embodiment of the present invention; and FIG. 4A- 4B is a flow chart of a webpage data acquisition method according to a fourth embodiment of the present invention.

The contents of the present invention will be explained by way of embodiments. It should be noted that the embodiments of the present invention are not intended to limit the invention as required by any of the embodiments. A specific environment, application or special way can be implemented. Therefore, the description of the embodiments is only for the purpose of illustrating the invention, and is not intended to limit the invention. In addition, in the following embodiments and drawings, elements that are not directly related to the present invention have been omitted and are not shown, and the dimensional relationships between the elements in the following figures are merely for ease of understanding and are not intended to be limiting. Actual ratio.

Please refer to Figures 1A~1B. 1A is a block diagram of a webpage data capture operation of the first embodiment of the present invention, and FIG. 1B is a block diagram of a webpage data capture device 1 of the first embodiment of the present invention. The webpage data capture device 1 includes a receiving unit 11 and a processing unit 13, and is connected to a web server 9 via the receiving unit 11. The interaction between components will be further elaborated below.

First, when it is necessary to analyze the web page of the web server 9, the receiving unit 11 of the web data extracting device 1 receives the plurality of web materials wp from the web server 9. Among them, based on the principle of Internet usage, each web page data wp has its corresponding uniform resource locator (URL) ul.

Next, the processing unit 13 of the webpage data retrieval device 1 divides the plurality of webpage data wp into at least one URL group ug according to the address relevance of the plural URL ul of the plurality of webpage data wp. The at least one URL group u1 includes a first URL group UL1, and the first URL group UL1 includes at least part of the web page data wp.

It should be noted that the purpose of this grouping is to initially classify webpages with higher similarity of webpage content according to the URL characteristics, so as to facilitate subsequent comparison analysis. In other words, since the URLs of the same board and typesetting webpages are usually similar in form, the initial grouping can be performed according to the address relevance of the URL of the webpage data.

Then, the processing unit 13 of the webpage data retrieval device 1 selects a first webpage data WP1 and a second webpage data WP2 from a part of the webpage data of the first URL group UL1, and parses the first webpage data WP1 and the second The web page data WP2 has a web node data set wpg.

In detail, since a single webpage includes a plurality of webpage nodes, the syntax of the first webpage data WP1 and the second webpage data WP2 can be analyzed to obtain a webpage node dataset wpg including a plurality of webpage node materials ND. The web page node data ND includes one of an XML path language (NDX) and a text content NDT.

According to this, the processing unit 13 of the webpage data extraction device 1 can select the webpage node according to the path association of the plural XML path language NDX of the plurality of webpage node data ND of the webpage node data set wpg and the textual relevance of the plural text content NDT. The data set node data ND of the data set wpg is divided into a plurality of web page node data groups ndg. The webpage node data group ndg includes at least part of the webpage node data ND.

It should be noted that, similarly, this grouping means that, according to the characteristics of the XML grammar and the text content, the webpage nodes with higher content similarity are classified to facilitate the judgment of the subsequent main content. In other words, according to the path association of the XML path language of the webpage node, the webpage nodes with high XML syntax similarity are grouped, and on the other hand, the content similarity is higher according to the textual relevance of the text content of the webpage node. The web nodes are grouped.

Next, the processing unit 13 of the webpage data extraction device 1 calculates each The sum of the text content of one of the webpage node data ND of the webpage node data group ndg (not shown), that is, the total length of the text of the webpage node data ND of the same webpage node data group ndg, and judging the plural according to the sum of the plural text contents At least one primary web node data group MNDG of the web node data group ndg.

Specifically, since the webpage node data having the main content in the same webpage usually has a large amount of text content, the foregoing grouping is mainly based on the sum of the text contents of the webpage node data of the same webpage node data group. The web node data with the main content is divided with the web node data without the main content.

According to this, the processing unit 13 of the webpage data retrieval device 1 can determine a webpage main content capture information MX according to the XML path language NDX of the partial webpage node data ND included in at least one main webpage node data group MNDG. Furthermore, the main content of the web page capture information MX is mainly a collection of XML path language NDX.

In this case, in the case that the foregoing URL group has the same nature (for example, template and typesetting) webpage, the processing unit 13 of the webpage data extracting apparatus 1 can directly follow the set of the XML path language NDX in the URL group. Directly select the web page node with the main content, and analyze and utilize the subsequent main content.

Please refer to Figure 2A-2B. 2A is a block diagram of a webpage data capture operation of the second embodiment of the present invention, and FIG. 2B is a block diagram of a webpage data capture device 2 of the second embodiment of the present invention. The webpage data capture device 2 includes a receiving unit 21 and a processing unit 23, and is connected to the web server 9 via the receiving unit 21. First The second embodiment mainly uses the example to explain the webpage data capturing device 2 to extract the details of the analysis webpage.

Similarly, when it is necessary to analyze the webpage of the webpage server 9, the receiving unit 21 of the webpage data extracting apparatus 2 receives the plurality of webpage materials wp from the webpage server 9, and based on the internet usage principle, each webpage data wp has its Corresponding URL ul, wherein the webpage data wp and the corresponding URL ul are as follows:

Next, the processing unit 23 of the webpage data retrieval device 2 divides the plurality of webpage data wp into at least one URL group ug according to the address relevance of the plural URL ul of the plurality of webpage materials wp. The at least one URL group ul includes the first URL group UL1, and the first URL group UL1 includes at least part of the web page data WP. It should be noted that in the second embodiment, the URL grouping here is mainly based on the Minimum Edit Distance (MED).

In detail, the processing unit 23 of the webpage data retrieval device 2 calculates the minimum edit distance of the plurality of URLs ul of the plurality of webpage data wp, and the results are as follows:

Accordingly, the processing unit 23 of the webpage data retrieval device 2 can add the webpage data with the MED value less than a URL threshold to the same URL group according to the content of the above table. In the second embodiment, the URL threshold is 2, so the web page pair with the MED value of 1 will be grouped in the same URL group.

In detail, at least part of the webpage data WP included in the first URL group UL1 is http://www.aaaaa.com/item1~3.html . In addition, at least one URL group ul may also include a second URL group (not shown), and the second URL group includes at least part of the web page data WP, ie, http://www.aaaaa.com/list1~2 .html , but the operation of the same URL group is the same, and the subsequent will only be based on the first URL group UL1.

Then, the processing unit 23 of the webpage data retrieval device 2 selects the first webpage data WP1 with the highest amount of data (ie, the HTML size of the webpage data) and the second highest amount of data from the partial webpage data of the first URL group UL1. The second webpage data WP2, and parsing the webpage node data set wpg of the first webpage data WP1 and the second webpage data WP2.

In detail, since a single webpage includes a plurality of webpage nodes, the syntax of the first webpage data WP1 and the second webpage data WP2 can be parsed to obtain a webpage node dataset wpg including a plurality of webpage node materials ND. Among them, each web page node data ND includes a corresponding XML path language NDX and text content NDT, and the contents are as follows:

Subsequently, in the second embodiment, the duplicated or invalid webpage node data ND may be further deleted from the webpage node dataset wpg. Specifically, the processing unit 23 of the webpage data retrieval device 2 selects at least one invalid text content and at least one duplicate node data from the text content NDT according to the above table. Taking the above table as an example, the invalid text content is '0' and 'null', and the duplicate node data is 'html/body/div[1]/div[2]/div[2]/div[3]/div[3 ]/div[6]∥Return to the home page'. Therefore, the adjusted ND content of the web node data set wpg of the web node data set is as follows:

Then, the processing unit 23 of the webpage data extraction device 2 can obtain the webpage node data according to the path association of the plural XML path language NDX of the plurality of webpage node data ND of the webpage node data set wpg and the textual relevance of the plural text content NDT. The plurality of web page node data ND of the collection wpg is divided into a plurality of web page node data groups ndg.

In more detail, in the second embodiment, the technology of the webpage node data grouping here can be mainly divided into two parts. First, in the first part, similarly, the XML path language NDX of the webpage node data ND of the foregoing table is subjected to minimum edit distance calculation, and the web node data ND pair with the MED value less than an XML threshold value (not shown) is added. To the same path group XG. In the second embodiment, the grouping status is as shown in the following table:

Then, in the second part, in each path group XG, the TF-IDF (term frequency-inverse document frequency) calculation is performed on the text content NDT of the webpage node data ND, and the corresponding plural frequency frequency vector is obtained, and the two-word text is calculated. If the cosine value between the content frequency vectors of the content is greater than a text content threshold (not shown), it is added to the same web node data group ndg. In the second embodiment, the grouping status is as shown in the following table:

In this way, by integrating the above two parts of the grouping method, the web node data group ndg is formed, as shown in the following table:

It should be noted that the technique of using the keyword to perform the TF-IDF calculation of the correlation vector for the text content and calculating the cosine value of the two vectors to determine the correlation between the vectors should be easily understood by those skilled in the art through the prior art. The content is not described here, and the present invention is mainly used for the relevance basis of the grouping.

Then, the processing unit 23 of the webpage data extraction device 2 calculates the total text content of the webpage node data ND of each webpage node data group ndg, that is, calculates the total length of the text of the webpage node data ND of the same webpage node data group ndg. , as detailed below:

Next, the processing unit 23 of the webpage data extracting device 2 sorts the sum of the text contents corresponding to the different webpage node data groups ndg into a text content sum sequence, as shown in the following table:

Subsequently, the processing unit 23 of the web page data extracting device 2 calculates the difference between the sum of the adjacent text contents in the sorted text sum sequence: 1, 2, 1, 44, 1, and selects the maximum difference, that is, 44. Accordingly, since the webpage node data having the main content usually has a large amount of text content in the same webpage, the maximum difference occurs, that is, the webpage node data of the main content does not have The boundary between the main content of the web page node data.

Therefore, the processing unit 23 of the webpage data extraction device 2 can divide the text content sum sequence into a main area and a primary area according to the maximum difference, and determine at least one main page of the webpage node data group ndg according to the main area. The web node data group MNDG is shown in the following table:

Therefore, in the second embodiment, the XML path language NDX of the partial webpage node data ND included in the main webpage node data group MNDG is as follows:

Then, the processing unit 23 of the webpage data retrieval device 2 can perform the longest common subsequence algorithm for the XML path language NDX of the partial webpage node data ND included in the main webpage node data group MNDG, and determine the webpage. The main content captures information MX. In the second embodiment, the main content capture information MX of the webpage is: 'html/body/div[1]/main[1]/article[[0-9]+].*'.

In this way, in the case that the aforementioned URL group (ie, http://www.aaaaa.com/item1~3.html ) has the same nature (for example, template and typesetting) webpage, the processing unit of the webpage data capturing device 2 23 Subsequent selection of the web page node with the same main content capture information MX (ie html/body/div[1]/main[1]/article[[0-9]+].*), subsequent main content Analysis and utilization.

The third embodiment of the present invention is a webpage data extraction method, and the flowchart thereof is referred to FIG. The method of the third embodiment is applied to a web page data extracting device (for example, the web page data extracting device 1 of the foregoing embodiment). The webpage data retrieval device receives the plurality of webpage data from a webpage server. The detailed steps of the third embodiment are as follows.

First, step 301 is executed to enable the webpage data retrieval device to divide the plurality of webpage data into at least one URL group according to the address association of the plural URL of the plurality of webpage materials. The at least one URL group includes a first URL group, and the first URL group includes at least part of the webpage data. Step 302 is executed to enable the webpage data retrieval device to select a first webpage material and a second webpage data from a portion of the webpage data of the first URL group.

Step 303 is executed to enable the webpage data capture device to parse the first webpage data and the second webpage data to obtain a webpage node dataset. The webpage node data set includes a plurality of webpage node materials, and each webpage node data includes a corresponding one of an XML path language and a text content.

Step 304 is executed to enable the webpage data extraction device to divide the plurality of webpage node data of the webpage node data collection according to the path relevance of the XML path language of the webpage node data of the webpage node data collection and the textual relevance of the text content. Multiple web page node data groups. The webpage data group includes at least part of the webpage node data.

Step 305 is executed to enable the webpage data extraction device to calculate the sum of the text contents of one of the webpage node data of each webpage node data group. Step 306 is executed to enable the webpage data retrieval device to determine at least one major webpage node data group of the plurality of webpage node data groups according to the sum of the plurality of textual content. Finally, step 307 is executed to enable the webpage data retrieval device to determine the main content capture information of a webpage according to the XML path language of the webpage node data included in the at least one main webpage node data group.

The fourth embodiment of the present invention is a webpage data extraction method, and the flowchart thereof is referred to FIG. The method of the fourth embodiment is applied to a web page data extracting device (for example, the web page data extracting device 2 of the foregoing embodiment). The webpage data retrieval device receives the plurality of webpage data from a webpage server. The detailed steps of the fourth embodiment are as follows.

First, step 401 is executed to enable the webpage data retrieval device to divide the plurality of webpage data into at least one URL group according to the address relevance of the plural URL of the plurality of webpage materials. At least one URL group includes a first URL group, the first URL group includes at least part of the webpage data, and the minimum edit distance between the URLs of the partial webpage data in the first URL group is less than a URL threshold. .

Step 402 is executed to enable the webpage data retrieval device to select one of the first webpage materials with the highest amount of data and the second webpage data of the second highest data volume from the webpage data of the first URL group. Step 403 is executed to enable the webpage data capture device Parsing the first webpage data and the second webpage data to obtain a webpage node data set. The webpage node data set includes a plurality of webpage node materials, and each webpage node data includes a corresponding one of an XML path language and a text content.

Step 404 is executed to enable the webpage data extraction device to select at least one invalid text content and at least one duplicate node data from the text content, and the webpage node corresponding to the at least one invalid text content and the at least one duplicate node data from the webpage node data Remove from the collection.

Step 405 is executed to enable the webpage data retrieval device to divide the plurality of webpage node data of the webpage node data collection into a plurality of path groups according to the path association of the XML path language of the plurality of webpage node data of the webpage node data collection. The minimum edit distance between the XML path languages of the partial web node data of each path group is less than an XML threshold.

Step 406 is executed to enable the webpage data retrieval device to divide each path group into a plurality of webpage node data groups according to the textual relevance of the text content of the partial webpage node data for each path group. Wherein, in each path group, each text content of the part of the webpage node data has a term frequency vector, and in each path group, the cosine frequency between the text frequency of the text content of the part of the webpage node data group The value is greater than a text content threshold.

Step 407 is executed to enable the webpage data extracting device to sort the sum of the plurality of text contents into a sum of text content sums. Step 408 is executed to enable the webpage data retrieval device to calculate a complex difference between the sum of adjacent text contents in the sum of the text contents. Step 409 is executed to enable the webpage data capture device to select one of the maximum differences of the complex differences. Hold In step 410, the webpage data extracting device divides the text content sum sequence into a main area and a primary area according to the maximum difference.

Step 411 is executed to enable the webpage data retrieval device to determine at least one major webpage node data group of the plurality of webpage node data groups according to the main area. Step 412 is executed to enable the webpage data retrieval device to perform the longest common subsequence algorithm for the XML path language of the partial webpage node data included in the at least one main webpage node data group. Step 413 is executed to enable the webpage data retrieval device to determine the main content of the webpage to retrieve the information according to the result of step 412.

In summary, the webpage data extracting apparatus and the webpage data extracting method thereof of the present invention can automatically analyze the syntax of the template and typesetting of different webpage groups, and automatically find the webpage node having the main content. In this way, the webpage data can be retrieved more efficiently, and the subsequent relevant data analysis can be performed.

The above-described embodiments are merely illustrative of the embodiments of the present invention and the technical features of the present invention are not intended to limit the scope of the present invention. It is intended that any changes or equivalents of the invention may be made by those skilled in the art. The scope of the invention should be determined by the scope of the claims.

301~307‧‧‧Steps

Claims (14)

A webpage data capture method for a webpage data capture device, the webpage data capture device receives a plurality of webpage data from a webpage server, and the webpage data capture method comprises: (a) causing the webpage data capture apparatus to Addressing the address of the uniform resource locator (URL) of the webpage data, and dividing the webpage data into at least one URL group, wherein the at least one URL group includes a first URL group a group, the first URL group includes at least a portion of the webpage data; (b) causing the webpage data retrieval device to select a first webpage material and a first page from a portion of the webpage data of the first URL group (b) causing the webpage data capture device to parse the first webpage data and the second webpage data to obtain a webpage node data set, wherein the webpage node dataset includes a plurality of webpage node data, each webpage node The data includes a corresponding XML Path Language and a text content; (d) the web page data extraction device according to the web node node data collection of the webpage nodes The path association of the XML path language of the data and the textual relevance of the text content, the webpage node data of the webpage node data set is divided into a plurality of webpage node data groups, wherein each webpage node data group The group includes at least a portion of the webpage node data; (e) causing the webpage data capture device to calculate a sum of textual content of one of the webpage node data of each of the webpage node data groups; (f) extracting the webpage data Determining, according to the sum of the text contents, at least one main webpage node data group of the webpage node data group; (g) causing the webpage data extracting apparatus to according to the at least one main webpage node data grouping package The XML path language of the data of the web page nodes, which determine the content of the web pages, determines the main content of a web page to capture information. The method for extracting webpage data according to claim 1, wherein a minimum edit distance (Minimum Edit Distance) between the URLs of the webpage data in the first URL group is less than a URL threshold. The method for extracting webpage data according to claim 1, wherein the step (b) further comprises: (b1) causing the webpage data retrieval device to select a data amount from a portion of the webpage data of the first URL group. The highest first page of the webpage and the second webpage of the second highest amount of data. The method for extracting webpage data according to claim 1, wherein the step (c) further comprises: (c1) causing the webpage data extracting device to select at least one invalid text content and at least one repeating node from the text content. And deleting the webpage node corresponding to the at least one invalid text content and the at least one duplicate node data from the webpage node data set. The method of extracting webpage data according to claim 1, wherein the step (d) further comprises: (d1) causing the webpage data retrieval device to use the XML path language of the webpage node data according to the webpage node data collection. Path association, the webpage node data of the webpage node data set is divided into a plurality of path groups, wherein a minimum edit distance between the XML path languages of the webpage node data of each part of the path group is less than An XML threshold value; (d2) causing the webpage data capture device to divide each of the path groups into the webpages according to the textual relevance of the textual content of the webpage node data for each of the path groups Node data group; Wherein, in each of the path groups, each of the text content of the webpage node data has a term frequency vector; wherein, in each of the path groups, each of the webpage node data groups is part of the webpage node data The cosine values between the frequency frequencies of the words of the text content are greater than a text content threshold. The method for extracting webpage data according to claim 1, wherein the step (f) further comprises: (f1) causing the webpage data extracting device to sort the sum of the text contents into a sum of text content; (f2) The webpage data extraction device calculates a complex difference between the sum of adjacent text contents in the summation sequence of the text content; (f3) causing the webpage data extraction device to select a maximum difference of one of the differences; (f4) making the webpage data The capturing device divides the text content sum sequence into a main area and a primary area according to the maximum difference; (f5) causing the webpage data extracting device to determine the webpage node data group according to the main area At least one primary web node data group. The method for extracting webpage data according to claim 1, wherein the step (g) further comprises: (g1) causing the webpage data capture device to include a portion of the webpage node data included in the at least one main webpage data group The XML path language performs a longest common subsequence algorithm; (g2) causes the web page data extracting device to determine the main content of the web page to retrieve information according to the result of the step (g1). A webpage data capture device comprising: a receiving unit, configured to receive a plurality of webpage data from a webpage server; and a processing unit configured to: according to the address association of a plurality of uniform resource locators (URLs) of the webpage data, And the webpage data is divided into at least one URL group, wherein the at least one URL group comprises a first URL group, the first URL group includes at least part of the webpage materials; and the part of the first URL group Selecting a first webpage data and a second webpage data in the webpage data; parsing the first webpage data and the second webpage data to obtain a webpage node dataset, wherein the webpage node dataset includes a plurality of webpage node data Each web page node data includes a corresponding one of an XML path language and a text content; a path association of the XML path languages of the web page node data according to the web page node data set and the text content Text association, the webpage node data of the webpage node data collection is divided into a plurality of webpage node data groups, wherein each of the The webpage node data group includes at least some of the webpage node data; respectively, calculating a sum of text content of one of the webpage node data of each webpage node data group; and determining the webpage node data group according to the sum of the textual content At least one main webpage node data group; determining the main content of the webpage to retrieve information according to the XML path language of the webpage node data included in the at least one main webpage data group. The webpage data capture device of claim 8, wherein a minimum edit distance (Minimum Edit Distance) between the URLs of the webpage data of the first URL group is less than a URL threshold. The webpage data retrieval device of claim 8, wherein the processing unit is further configured to: select the first webpage data and the amount of data with the highest amount of data from a portion of the webpage data of the first URL group The second highest webpage data. The webpage data retrieval device of claim 8, wherein the processing unit is further configured to: select at least one invalid text content and at least one duplicate node data from the text content, and invalidate corresponding to the at least one The text content and at least one web node of the repeated node data are deleted from the web node data collection. The webpage data retrieval device of claim 8, wherein the processing unit is further configured to: according to the path association of the XML path languages of the webpage node data of the webpage node data collection, the webpage node data The collection of the webpage node data is divided into a plurality of path groups, wherein a minimum edit distance between the XML path languages of the webpage node data of each of the path groups is less than an XML threshold value; a group, according to the textual relevance of the text content of the webpage node data, the path group is divided into the webpage node data groups; wherein, among the path groups, some of the webpage nodes Each text content of the data has a term frequency vector; wherein, in each of the path groups, a cosine value between the word frequency vectors of the text content of the webpage node data of the webpage node data group Greater than a text content threshold. The webpage data extraction device of claim 8, wherein the processing unit is further configured to: sort the sum of the text contents into a sum of text content sum; calculate a complex difference of the sum of adjacent text contents in the summation sequence of the text content a maximum difference value of one of the difference values; the sum of the text content sum is divided into a main area and a primary area according to the maximum difference; and the web area data group is determined according to the main area At least one primary web node data group. The webpage data capture device of claim 8, wherein the processing unit is further configured to: perform the longest common for the XML path language of the webpage node data of the at least one main webpage data group The Longest Common Subsequence algorithm; determines the main content of the web page to capture information based on the results of the longest common subsequence algorithm.