TWI738126B - Web content filtering method - Google Patents

Abstract

A web content filtering method includes: S10: browsing a web; S20: cluster identifying a content of a home of the web; S30: finding out an article list in the home, the article list includes a plurality of URL link; S40: taking out two URL link in the article list; S50: respectively linking to the URL link and getting two article pages, the article page includes a plurality of data field; S60: comparing two article pages and removing the data field having same content; S70: marking remaining the data field into a collecting field; and S80: capturing the data of the collecting field.

Description

Methods of Web Content Screening

A method of data screening, especially a method of web content screening

Crawler software (web crawler) is a commonly used data collection tool nowadays. It is a tool that can automatically browse web pages, download and collect web content. Because it can replace traditional manpower and under the trend of technological development, crawler software plays an important role in the big data industry.

However, although crawling software can automatically retrieve web content, it cannot distinguish the collected data. Invalid data or advertisements still account for the majority. Therefore, the currently collected data are mostly screened by manual identification, or additional algorithms are designed to identify these data. This is not convenient for some specific analysis fields, and it also affects the accuracy of the analysis results.

Therefore, how to improve the accuracy of data collection of crawler software is worth considering for those with ordinary knowledge in the field.

The present invention provides a method for screening web content, including: S10: visiting a website; S20: grouping and identifying the content of the homepage of one of the websites; S30: finding an article list from the homepage, and the article list includes multiple URL links; S40: Take out two URL links from the article list; S50: Link to the URL link respectively to obtain two article webpages, which include multiple data fields; S60: compare two article webpages, and remove the data field with the same content; S70: change the remaining data fields The data field is marked as a collection field; and S80: Retrieve the data in the collection field.

In the above method for screening web content, the website is a news website.

In the above method for screening web content, in step S10, the Meta tag of the website is used to determine whether the website is a news website.

The above-mentioned method for screening web content further includes: S90: browsing the article webpage with the same location; and S100: capturing data in the collection field of the article webpages.

In the above method for screening webpage content, in step S90, it is determined whether the article webpage has the same position based on the Xpath position of the article webpage.

In the above method for screening web content, in step S20, the website is grouped by means of mechanical learning.

In the above-mentioned method for screening web content, a data collection system implements a method for screening the web content.

S10~S100: Flowchart steps

110: Home

111: category field

112: ad slot

113: Headline column

114: Article list

1411, 1411a~d: URL link

120: Article page

121: News Classification

122: News Title

123, 125: Advertising slot

124, 126: News main article

200: Data Collection System

210: Browse Module

220: Judgment Module

230: contrast module

240: Collection module

250: database

FIG. 1A shows the method for screening web content of the present invention.

Figure 1B shows a data collection system.

Figure 2 shows a schematic diagram of the home page.

Figure 3 shows a schematic diagram of the article webpage.

Figure 4 shows a schematic diagram of removing data fields.

The present invention provides a method for screening webpage content, which can correctly extract main content from webpages, avoid wrong or invalid content, and improve the accuracy of data collection.

In this embodiment, a method of filtering web content is performed through a data collection system 200. Please refer to Figure 1B. Figure 1B shows a data collection system. The data collection system 200 is, for example, a crawler software. The data collection system 200 includes a browsing module 210, a judgment module 220, a comparison module 230, a collection module 240, and a database 250.

The data collection system 200 is connected to a plurality of websites 100 via the browsing module 210 for browsing. The browsing module 210 judges the website category according to the Meta tag of the website 100, preferably browsing news websites. The judgment module 220 is adapted to judge the home page 110 in the website 100 and find a specific location from the home page 110. In this embodiment, the judgment module 220 is a judgment model that has undergone mechanical training or deep learning training. Therefore, it can identify the content block on the home page 110 from the home page 110 by itself, find the article list 114 therefrom, and find the article list 114 from the article list 114. Get two URL links in. The browsing module 210 will browse the article webpage 120 corresponding to the two URL links.

The comparison module 230 is adapted to compare the article webpage 120. After the two article webpages 120 are browsed through the browsing module 210, the comparison module 230 compares the content of the two article webpages 120. After the comparison, the same fields in the two article web pages 120 are removed, and fields with different contents are retained. And define the reserved field as a collection field.

The collection module 240 is suitable for collecting data in the article webpage 120, especially collecting data in the collection field. Further, the data collection system 200 determines the collection fields of the website 100 through the comparison module 230, and the collection module 240 also collects data in the collection fields of other article webpages 120 in the website 100. The collected information is sent to the database 250 for storage.

The following will explain the specific method of web content screening. Please refer to FIG. 1A. FIG. 1A illustrates the method for screening web content of the present invention. First, visit a website 100 (step S10). In a preferred embodiment, the website is a news website, and the Meta tag of the website can be used to determine whether the website 100 is a news website. After visiting the website 100, the content on the home page 110 of the website 100 is clustered and identified (step S20).

Please refer to Figure 2, which is a schematic diagram of the home page. In the embodiment of FIG. 2, the home page 110 is, for example, the home page 110 of a news website. Therefore, the home page 110 includes a category field 111, an advertisement space 112, a headline field 113, and an article list 114. In step S20, the category column 111, the advertisement slot 112, the headline column 113 and the article list 114 are grouped and identified.

After the content of the homepage 110 is grouped, the article list 114 in the homepage 110 is found (step S30). The article list 114 has a plurality of URL links 1141, and the URL links 1141 are connected to the article webpage 120 in the website. At this time, two of the URL links 1141 are extracted from the article list 114 (step S40). In the embodiment of FIG. 2, for example, the URL links 1141a and 1141b. Then, link to the web pages corresponding to the URL links 1141a and 1141b, respectively, to obtain two article web pages 120 (step S50).

Please refer to FIG. 3, which is a schematic diagram of the article webpage. The article webpage 120 will include multiple data fields, and the data fields are fields for the content recorded on the webpage. In this embodiment, the data fields include fields such as news category 121, news headline 122, advertisement positions 123, 125, news main text 124, 126, and so on.

In step S50, two article webpages 120 are obtained, so the data fields of the two article webpages 120 are then compared, and the data fields with the same content are removed. Please refer to FIG. 3 and FIG. 4. FIG. 4 shows a schematic diagram of removing data fields.

In this embodiment, since the data collection system 200 is used to browse webpages, and the current network technology is used, the webpages browsed by the same terminal will have the same data fields. For example, the news category 121 is a website for users to select news categories, so it is unchanged. Advertisement slots 123 and 125 are because the advertisement content is usually placed when the browser renders the webpage, and the data collection system 200 is not used to browse the webpage interface. It is not a browser, so the advertisement slots 123 and 125 are a string of code representing loading advertisements. Therefore, the advertisement slots 123 and 125 also have the same content for the data collection system 200.

In summary, after the comparison, the data fields with the same content, that is, the news category 121 and the advertisement positions 123 and 125 can be removed. What remains is the data field that records the news content, such as the news headline 122 and the news main text 124, 126.

Next, mark the retained data field as a collection field (step S70), and retrieve the data in the collection field (step S80). Through step S10 to step S80, the article configuration in the website can be established, for example, the positions of the news headline 122 and the news main text 124, 126, and the position of the data field can be applied to collect other articles in the website. In other words, the main purpose of step S10 to step S80 is to confirm the article structure of the website 100, find the location of the collected subject matter, and exclude the location of data that does not need to be collected. When other websites need to be collected, steps S10 to S80 are executed on another website to confirm the article structure of the other website.

Therefore, next browse the article webpage 120 with the same position (step S90). In this embodiment, the Xpath position of the article webpage 120 is used to determine whether the article webpage 120 has the same position. That is, these article pages belong to the same website and also have the same article configuration. After that, the data in the collection fields in the article webpages can be retrieved (step S100).

Through the above steps S10 to S80, extract the two webpages from the same website and compare the content of the webpages to determine the data fields that need to be collected (such as news main text, title, publication time, etc.), and remove those that do not need to be collected Data fields (such as advertisements, news categories, etc.). After the judgment of step S10 to step S80, the article configuration under the same website can be judged. After that, step S90 to step S100 are performed to further collect the content of other articles under the same website. By repeatedly performing steps S10 to S100, webpage data can be collected more efficiently and invalid data can be eliminated.

The description of the present invention is as above, but it is not intended to limit the scope of the patent rights claimed in this creation. The scope of its patent protection shall be determined by the scope of the attached patent application and its equivalent fields. Anyone who has common knowledge in the field Those who know, without departing from the spirit or scope of this patent, any changes or modifications made are equivalent changes or designs completed under the spirit of this creation, and should be included in the scope of the following patent applications.

S10~S100: Flowchart steps

Claims (7)

A method for screening webpage content includes: S10: visiting a website; S20: grouping and identifying the content of the homepage of one of the websites; S30: finding an article list from the homepage, and the article list includes multiple URL links ; S40: fetch two links to the website from the article list; S50: link to the website links respectively to obtain two article webpages, the article webpage includes multiple data fields; S60: compare two article webpages, Remove the data field with the same content; S70: mark the remaining data field as a collection field; and S80: retrieve the data in the collection field. Such as the method for screening web content described in item 1 of the scope of patent application, wherein the website is a news website. For example, in the method for screening web content described in item 2 of the scope of patent application, in step S10, the Meta tag of the website is used to determine whether the website is a news website. For example, the method for screening web content described in the first item of the scope of the patent application further includes: S90: browsing the article webpage with the same location; and S100: extracting the data in the collection field of the article webpages. For the method for screening web content described in item 4 of the scope of patent application, in step S90, it is determined whether the article webpage has the same position based on the Xpath position of the article webpage. As described in the first item of the scope of patent application, in the method for screening web content, in step S20, the website is grouped by means of mechanical learning. For example, the method for screening web content described in items 1 to 6 of the scope of patent application is a method of performing the screening of the web content by a data collection system.

Images