CN104598536A - Structured processing method of distributed network information - Google Patents

Abstract

The invention discloses a method for structured processing of distributed network information. Configure the network information collection task, classify and save the webpages that users are interested in, and use them as target webpages; collect network information, collect webpages through multiple map/reduce processes and perform structured processing, and save them in the HDFS file system Middle; use the tree edit distance method to carry out structured clustering on the structured web pages; carry out structured extraction on the clustered web page information and save them in the database. The present invention adopts a distributed architecture, uses the computing and storage capabilities of cheap computer clusters to process network data with a huge amount of data; effectively classifies web pages; adopts a structured method to extract and store network information, which is convenient further analysis and processing of network information.

Description

A Distributed Network Information Structural Processing Method

technical field

The invention relates to a network information processing method in the field of network information collection, in particular to a structured collection and processing method of distributed network information.

Background technique

A distributed system is a system that efficiently organizes cheap computing clusters to perform large-scale data operations and storage.

Distributed systems are different from stand-alone systems. The use of computer clusters for data calculation and storage needs to balance the cost of single-node computing power and inter-node communication. At the same time, the system availability and data recovery caused by node failures in the cluster must also be considered. issues such as sex. Hadoop distributed processing and HDFS distributed file system are open source distributed computing and storage systems designed and developed based on the Map/Reduce computing model proposed by Google. Because of its effective solution to the problems in distributed systems and the simplicity and generality of its architecture, it has been widely used in many fields.

The structured clustering method is one of the clustering methods. Different from the content-based clustering method, the structured clustering method emphasizes clustering according to the structure, which requires different similarity measurement methods. The tree edit distance method is a method to measure the similarity of the tree structure. Converting one tree to another means performing a series of node insertion, deletion and replacement between the two trees. Each operation costs A certain cost. If the structure difference of the two trees is large, it means that the operation cost is high, and if the operation cost is low, it means that the structure difference of the trees is small. Therefore, the edit distance of the tree represents the minimum cost required for the transformation of two trees.

Network information collection is usually also called web crawler, which is widely used in Internet search engines or other similar websites to obtain or update the content and retrieval methods of these websites. It can automatically collect all the page content it can access for further processing by search engines and the like.

The information captured by existing web crawlers is stored in the storage system in the form of original web pages. Such a storage method has the following disadvantages. First, storing in the form of the original web page requires a large storage space; second, there are a large amount of irrelevant information in the stored information, such as advertisements; It is a semi-structured method. Compared with the structured storage method, the semi-structured storage method will cause certain obstacles to the use of further information.

Contents of the invention

The object of the present invention is to provide a method for structured acquisition and processing of distributed network information aiming at the deficiencies of the existing network information acquisition technology.

The technical scheme adopted in the present invention comprises the following steps:

1) Configure the network information collection task, classify and save the webpages that the user is interested in, and use them as target webpages;

2) Collect network information, collect web pages through multiple map/reduce processes and perform structured processing, and save them in the HDFS file system;

3) carrying out structured clustering on the structured processed web pages by means of tree edit distance;

4) Structurally extracting the clustered web page information and storing it in the database.

Described step 2) specifically comprises:

2.1) Obtain the URL seed file, save the URL seed file set in the folder to be captured in the HDFS file system, the folder to be captured contains URLs to be captured, and the initial layer sequence number is set to 1;

2.2) Determine whether the folder to be captured is empty, if so, jump to step 2.7); otherwise, proceed to the next step 2.3);

2.3) collect the corresponding webpages of each URL seed file in the HDFS file system through the map/reduce process, and store in the webpage storage folder in the HDFS file system, and the webpage storage folder stores unprocessed webpages;

2.4) Through the map/reduce process, extract and analyze the new URL from the captured webpage in the webpage storage folder, and save the new URL in the temporary folder of the HDFS file system. The temporary folder has been parsed out the URL;

2.5) Optimize the temporary folder through the map/reduce process, filter the URLs therein, remove the duplicate URLs, and then update the results in the folder to be captured in the HDFS file system;

2.6) Add layer number +1;

2.7) Judge the layer sequence number, if the current layer sequence number is greater than the capture depth value _Depth then enter step 2.7), otherwise jump to step 2.2);

2.7) Merge the multiple webpage storage folders obtained in the above steps into one webpage storage folder through the process of map/reduce, and remove duplicate webpages therein.

Described step 3) carries out clustering by map/reduce process, concrete steps are as follows:

3.1) In the map stage, for each webpage in the combined webpage storage folder obtained in step 2), the tree edit distance method is used to calculate the tag tree TREE _x of each webpage and each of the target webpages C _i The tree edit distance DIS _Ci between the tag tree TREE _Ci , get the tree edit distance set {DIS _C1 , DIS _C2 , DIS _C3 , ..., DIS _Cn }, and generate key-value pairs <C _i , WEB>, and then from the tree Select the minimum tree edit distance DIS _Cmin from the edit distance set, and pass the key-value pair <C _min , WEB> corresponding to the minimum tree edit distance DIS _Cmin to the reduce stage;

3.2) In the reduce stage, according to the above-mentioned key-value pair <C _i , WEB>, the webpages with the same key will be merged into a file DOC _Ci as the same type of webpage, and stored in the result folder of the HDFS file system, Each file DOC _Ci saves webpages with the same webpage structure, and obtains the structured webpage clustering results {DOC _C1 , DOC _C2 , DOC _C3 , . . . , DOC _Cn }, and completes the structured webpage clustering.

The step 4) extracts each type of webpage according to the structured webpage clustering result {DOC _C1 , DOC _C2 , DOC _C3 , ..., DOC _Cn } obtained in the step 3), and extracts the information in the webpage Save to the database.

Extracting the nodes of the corresponding tag tree in the webpage to the corresponding fields in the database.

Different extraction methods are used for the different types of web pages.

Step 1) configures the network information collection task, which is an interactive interface. Different from crawlers of search engines, the main purpose of the present invention is to monitor specific information sources on the network. As an information source, the web pages that users are interested in can be divided into text information, picture information, and video information according to content types, and can also be divided into news information and advertisement information according to content attributes. At the same time, the update frequency of different information sources is not the same. Through the configuration of network information collection tasks, the information of the collected information sources can be determined, so that different collection methods can be adopted for different types of information sources.

Step 2) Based on Hadoop distributed processing and HDFS distributed file system, the network collection tasks defined in step 1) are distributed and captured.

Step 3) performs structured clustering of webpages, and the webpages captured in step 2) are still stored in HDFS in the form of semi-structured files, and cannot be directly extracted into the database. Step 3) The web pages are clustered according to their structure by adopting the method of tree edit distance. Web pages are written in HTML, and a single HTML file can be abstracted into a tag tree. For web pages with the same information, their tag trees also have similar or even the same structure. In order to measure the similarity of tag trees, the present invention adopts the method of tree edit distance to classify the web pages captured in step 2).

Step 4) Extract the web page information in a structured manner, according to the result of structured clustering in step 3) and the extraction method for each type of web page in step 1), extract the information in the web page and save it in the database.

The beneficial effects that the present invention has are:

The present invention adopts a distributed architecture, utilizes the computing and storage capacity of cheap computer clusters to process network data with a huge amount of data; adopts the structured clustering method of webpages with tree editing distance, and effectively classifies webpages; adopts The network information is extracted and saved in a structured way, which facilitates further analysis and processing of the network information.

Description of drawings

Fig. 1 is a flowchart of the implementation steps of the present invention.

Fig. 2 is the web page tag tree in step 3.1) of the present invention.

Detailed ways

The present invention will be further described below in conjunction with drawings and embodiments.

As shown in Figure 1, the present invention comprises the following steps:

1) Configure the network information collection task, classify and save the webpages that the user is interested in, as the target webpage; obtain the target webpage set {C ₁ , C ₂ , C ₃ , ..., C _n used for clustering in the subsequent steps }; When the target webpage of the present invention is classified and saved, different information types of the same website can be classified. For example, the information types that may exist in the same website can be divided into news, product data, and pictures.

2) Collect network information, collect web pages through multiple map/reduce processes and perform structured processing, and save them in the HDFS file system.

Step 2) Based on Hadoop distributed processing and HDFS distributed file system, the network collection tasks defined in step 1 are distributed and captured.

2.1) Obtain the URL seed file, save the URL seed file set in the folder to be captured in the HDFS file system, the folder to be captured contains URLs to be captured, and the initial layer sequence number is set to 1;

2.2) Determine whether the folder to be captured is empty, if so, jump to step 2.7); otherwise, proceed to the next step 2.3);

2.3) collect the corresponding webpages of each URL seed file in the HDFS file system through the map/reduce process, and store in the webpage storage folder in the HDFS file system, and the webpage storage folder stores unprocessed webpages;

2.4) Extract and parse the new URL from the captured webpage in the webpage storage folder through the map/reduce process, and save the new URL in the temporary folder of the HDFS file system. The temporary folder stores the parsed Get the URL;

2.5) Optimize the temporary folder through the map/reduce process, filter the URLs therein, remove the duplicate URLs, and then update the results in the folder to be captured in the HDFS file system;

2.6) Add layer number +1;

2.7) Determine the layer number, if the current layer number is greater than the capture depth value _Depth , then enter step 2.7), otherwise jump to step 2.2), and then repeat the above steps 2) to 6) until the current layer number is greater than the capture depth value _Depth ;

2.7) Through the process of map/reduce, the multiple webpage storage folders obtained in the above steps are merged into one webpage storage folder according to the hash value of the webpage, and the repeated webpages are removed; the merged webpage storage folder can be saved in In the web page folder of the HDFS file system, this folder saves the captured web pages.

3) carrying out structured clustering on the structured processed web pages by means of tree edit distance;

Step 3) clustering through the map/reduce process, the specific steps are as follows:

3.1) In the map stage, for each webpage in the combined webpage storage folder obtained in step 2), such as a webpage of interest to the user in step 1), such as a webpage in a news webpage of a certain website, the map stage The target webpage and the captured webpage are extracted into a tag tree form, as shown in Figure 2. Using the tree edit distance method, calculate the tree edit distance DIS _Ci between the tag tree TREE _x of each web page and the tag tree TREE _Ci of each target web page C _i , and obtain the tree edit distance set {DIS _C1 , DIS _C2 , DIS _C3 ,..., DIS _Cn }, and generate a key-value pair <C _i , WEB>, then select the minimum tree edit distance DIS _Cmin from the tree edit distance collection, and set the key-value pair corresponding to the minimum tree edit distance DIS _Cmin < C _min , WEB> pass to the reduce stage;

3.2) In the reduce stage, according to the above-mentioned key-value pair <C _i , WEB>, the webpages with the same key will be merged into a file DOC _Ci as the same type of webpage, and stored in the result folder of the HDFS file system, The result folder contains files DOC _C1 , files DOC _C2 , files DOC _C3 , ..., files DOC _Cn , each file DOC _Ci saves web pages with the same web page structure, and obtains the structured web page clustering result {DOC _C1 , DOC _C2 , DOC _C3 ,..., DOC _Cn }, complete the structured clustering of web pages.

4) Structurally extracting the clustered web page information and storing it in the database.

Step 4) According to the structured webpage clustering results obtained in step 3) {DOC _C1 , DOC _C2 , DOC _C3 , ..., DOC _Cn }, extract each type of webpage, extract the information in the webpage and save it in the database In the extraction, the node of the corresponding label tree in the web page may be extracted to the corresponding field in the database.

Different types of webpages use different extraction methods, and the extraction method {R ₁ , R ₂ , R ₃ , ..., R _n } for each type of webpage can be defined to extract the information in the webpage and save it in the database.

Implementation examples of the present invention are as follows:

For an electric vehicle website, users need to obtain the news webpages and car model parameter webpages. For these two types of webpages, each type obtains a typical webpage as the target webpage, thus forming a set of target webpages {C ₁ , C ₂ }.

In step 2), distributed crawling is performed on the electric vehicle website to obtain its webpage data. The time-consuming process is related to the scale of the website to be crawled and the scale of the cluster that executes the crawling task. For a cluster of ten nodes, a fully loaded The crawling rate can reach 100,000 pieces/hour.

Step 3) The webpages obtained from the electric vehicle website are divided into two types of webpages: news and car model parameters through clustering, and the accuracy rate of clustering in this process can be more than 95%.

Step 4) carry out structural extraction for these two types of web pages, for news web pages, extract title, text, release date, source and other information, save in the database; for car model parameter class web pages, extract different model parameters, saved to the database.

Among them, an R _i form of an extraction method: wherein PubTitle, Content, PublicationDate, and DCSource are fields in the database, and each field is followed by a webpage xpath path, which defines the position of the field in the webpage. For each type of webpage file DOC _Ci , use its corresponding extraction method R _i to extract the corresponding data in the webpage to the corresponding field in the database.

It can be seen that during the entire crawling process, only the typical webpages of the two types of webpages that the user is interested in are required, and this method can extract and store the corresponding types of information in the target website in a structured manner and save them in the database. The process provides a faster information processing rate on the basis of maintaining a higher accuracy rate (above 95% obtained in the embodiment).

Therefore, based on the HDFS file system, the present invention proposes to use the tree edit distance to cluster webpages according to the webpage structure, and to classify each type of webpage according to the method based on the clustering results. The information is extracted and stored in the database, so as to realize the structured collection of web page information, which has a remarkable technical effect.

Claims (6)

1. a distributed network information structuring disposal route, is characterized in that the step of the method is as follows:

1) network information gathering task is configured, interested for user webpage is carried out classification and preserve, as target web;

2) network information is gathered, gather webpage by multiple map/reduce process collaborate and carry out structuring process, being kept in HDFS file system;

3) webpage after structuring process is adopted the mode of tree edit distance, carry out structuring cluster;

4) structuring extraction is carried out to the info web after cluster, be saved in database.

2. a kind of distributed network information structuring disposal route according to claim 1, is characterized in that: described step 2) specifically comprise:

2.1) obtain URL seed file, the set of URL seed file is saved to HDFS file system wait capture in file, wait that capturing file deposits the URL that will capture, and to arrange initial sequence number be 1;

2.2) judge that whether wait to capture in file is empty, if so, then jumps to step 2.7); Otherwise, carry out next step 2.3);

2.3) gathered by the webpage that map/reduce process is corresponding to each URL seed file in HDFS file system, and be kept at web storage file in HDFS file system and deposit, crude webpage deposited by web storage file;

2.4) webpage captured in web storage file is therefrom extracted to the URL resolving and make new advances by map/reduce process again, and be kept at by new URL in the temporary folder of HDFS file system, temporary folder deposits the URL parsed;

2.5) by map/reduce process optimization temporary folder, filter URL wherein, the URL of repetition is removed, then by result waiting to capture in file and upgrade in HDFS file system;

2.6) by sequence number+1;

2.7) sequence number is judged, if current sequence number is greater than capture depth value D _epththen enter step 2.7), otherwise jump to step 2.2);

2.7) a web storage file merged into by multiple web storage files above-mentioned steps obtained by the process of map/reduce, and removes the webpage wherein repeated.

3. a kind of distributed network information structuring disposal route according to claim 1, is characterized in that:

Described step 3) carry out cluster by map/reduce process, concrete steps are as follows:

3.1) in the map stage, for step 2) each webpage in web storage file after the merging that obtains, utilize tree edit distance method, calculate the tag tree TREE of each webpage respectively _xwith target web C described in each _itag tree TREE _cibetween tree edit distance DIS _ci, obtain tree edit distance set { DIS _c1, DIS _c2, DIS _c3..., DIS _cn, and generate key-value pair <C _i, WEB>, then chooses minimum tree editing distance DIS from tree edit distance set _cmin, by minimum tree editing distance DIS _cmincorresponding key-value pair <C _min, WEB> passes to the reduce stage;

3.2) in the reduce stage, according to above-mentioned key-value pair <C _i, in WEB>, the webpage with same keys is merged into a file DOC by key assignments _cimiddle as same class webpage, and be kept in the destination file folder of HDFS file system, each file DOC _cisave the webpage with same web page structure, obtain structuring website construction result { DOC _c1, DOC _c2, DOC _c3..., DOC _cn, complete the structuring cluster of webpage.

4. a kind of distributed network information structuring disposal route according to claim 1, is characterized in that: described step 4) according to step 3) in the structuring website construction result { DOC that obtains _c1, DOC _c2, DOC _c3..., DOC _cn, each class webpage is extracted, the information in webpage is extracted and is saved in database.

5. a kind of distributed network information structuring disposal route according to claim 4, is characterized in that: the field corresponding in database by the Node extraction of tag tree corresponding in described webpage.

6. a kind of distributed network information structuring disposal route according to claim 4, is characterized in that: described inhomogeneous webpage adopts different extracting modes.