CN104809231A - Mass web data mining method based on Hadoop - Google Patents

Abstract

The invention discloses a mass web data mining method based on Hadoop, and belongs to the field of computer data processing. A genetic algorithm is fused with the MapReduce of Hadoop, and mass Web data in a Hadoop-based distributed file storage system (HDFS) is mined to further verify the high efficiency of a platform, and a preferred access route of a user in a Web log is mined with a fused algorithm on the platform. As proved by an experiment result, the efficiency of Web data mining can be remarkably increased by processing of a large amount of Web data with a distributed algorithm in Hadoop.

Description

Mining data based web mass of Hadoop

FIELD

[0001] The present invention discloses a large-scale web data mining method, belonging to the field of computer data processing, in particular to a massive Hadoop web based data mining.

Background technique

[0002] For the current rapid growth of computing power of Web data size, a single node can no longer qualified for large-scale data analysis, in recent years, with the "cloud computing" the rise of technology, people will look to mass data storage and processing He turned to this emerging technology. Hadoop "cloud computing" platform, the biggest advantage is that it achieved a "near memory computing" thinking, the traditional "mobile data computing to close" mode overhead when the data reaches a mass scale is too large, and the "mobile computing to close stores" can save huge amounts of data transmission network of the large overhead, will be able to slashing processing time. With "cloud computing" the rise of technology, the existing data mining method and "cloud computing" platform integration to improve the efficiency of data mining, but the current study based on data mining is focused on improving the effectiveness of mining system, while ignoring the management of massive data processing speed. The present invention provides a web data mining method, the conventional genetic algorithm based fusion Hadoop mass of the MapReduce and Hadoop, massive excavation for Web data Hadoop distributed file storage system in the HDFS. To further validate the efficiency of the platform, digging preference access path to the Web user logs in using the algorithm of integration on this platform. Experimental results show that the use of algorithms in Hadoop distributed processing large amounts of Web data, Web data mining can significantly improve the efficiency, verify the availability of the system.

SUMMARY

[0003] The present invention is directed to the study based on data mining focused on improving the effectiveness of mining system, the neglect of the defect management processing speed mass data provided a large-scale web-based data mining Hadoop, the Hadoop the use of distributed algorithms handle large volumes of Web data, Web data mining can significantly improve the efficiency, verify the availability of the system.

[0004] The specific embodiment of the present invention is proposed:

Based on mass Hadoop web data mining methods:

Data mining environment structures: and selected to serve as NameNode MapReduce JobTracker servers in the server cluster, to calculate the remaining nodes and the data storage node, the test data set from the Web server log server room;

Data mining job submission: user submits a job based on the MapReduce programming specification written;

Task assignment: calculate the number of tasks required Map and Reduce the number of tasks, and the task to a sub Map TaskTracker execution node; while assigning respective TaskTracker Reduce task execution;

Task data reading: Map subtask is assigned to the node reads the already divided TaskTracker good data as input, after the process of generating the key / value pairs;

Map task execution: TaskTracker Map function call user-written to obtain from JobTracker, and intermediate results are cached in memory;

Write local intermediate results: the intermediate results in the memory reaches a certain threshold, TaskTracker written to the local disk;

Remote readout of the intermediate file JASReduce JobTracker TaskTracker obtained from the subtasks, sorted according to the key value of the position information by pulling data socket, and the intermediate result using the intermediate result, having the same key to be combined;

Reduce perform tasks: the task execution Reduce TaskTracker traversing the intermediate data of any sort, delivered to the user's Reduce function performed Reduce process;

Output: When all the Map tasks and Reduce tasks are completed, JobTracker control will result Reduce wrote on HDFS.

[0005] The process for the assigned tasks: JobTracker job control node according to the situation of the job, the number of tasks to calculate the Map and Reduce the number of tasks required, and the load distribution of data and a corresponding node, the task given to store the Map lightest load task and task execution node, at the request of the job results, assign the corresponding TaskTracker Reduce perform tasks.

[0006] The results of local write intermediate is the procedure: the intermediate results in the memory reaches a certain threshold, TaskTracker written to the local disk, the same data into these intermediate Reduce the number of tasks through the partition function of the partition, and send a place of their local disk to JobTracker, JobTracker then sends the location information to Reduce TaskTracker perform subtasks.

[0007] The assignment of the tasks is evaluated as: MapReduce framework automatically be divided into a set of test parts M, and the data format (<id, <A, B>), id represents the log number; B represents the user's current pages visited; a represents the user before accessing the page B to stay;

Map then input operation for each record scan, the data set is initialized according to the above format; Map after the operation, to obtain an intermediate result << A, B>, 1>, i.e., the user visited the page B from page A; Reduce operation will have the same page in accordance with the intermediate result <a, B> jump access mode are combined to obtain an output result << a, B>, n>, where, η representatives access path a -> Β frequency.

[0008] Each data set, respectively, to convert the result to Reduce operation list structure, the head of the linked list to save the value of k chain structure: k (A, B) (B, D) (D, E) ..., wherein, k represents the length of the chromosome; a, B, C, D, E representing the page.

[0009] each of said inner set of genetic data operation, until the k does not change, the operation ends.

Genetic operation process [0010] according to: randomly selecting parent chromosomes two chromosomes, and then generates a random insertion position lns, Del erasure position, indels length Len; Compares two chromosomes length, if equal, head and tail is determined whether there is an overlap, if coincident, generating new chromosomes connection, otherwise, not generated offspring chromosomes; if unequal, the determined insertion and deletion of gene segments 2 are the same, if the same, then merged into a chromosome as a new chromosome, otherwise, does not generate offspring chromosomes.

When the genetic generation [0011] or a multiple of 50, operation data between populations marriage.

[0012] the benefits of the present invention: The present invention relates to the genetic algorithm is MapReduce Hadoop fusion, massive excavation for Web data distributed file storage system in the HDFS Hadoop further validate the efficiency of the platform, the platform mining on the use of fusion algorithm preference access path, the experimental results show that Web users log in, the use of distributed algorithms in Hadoop processing large amounts of Web data, Web data mining can significantly improve efficiency.

BRIEF DESCRIPTION

[0013] FIG topology diagram of the data mining method of the invention a.

Detailed ways

[0014] DESCRIPTION OF THE DRAWINGS The present invention further.

[0015] - seed mass Hadoop web based data mining methods:

Data Mining build environment: Had00p six Pao PR2310N the internet servers, and wherein the HDFS NameNode JobTracker MapReduce is acting as a server by the remaining computing nodes and five act as a data storage node. Web server log server room of test data sets from ants Square software. Test program using Eclipse for Java developer platform for development;

① Data mining job submission: user submits a job based on the MapReduce programming specification written;

② Task Assignment: calculate the number of tasks required Map and Reduce the number of tasks M R, and the task to a sub Map TaskTracker execution node; TaskTracker performed while assigning respective Reduce task; specific process: JobTracker job control node according to the situation of the job, calculated Map and Reduce the number of tasks required number of tasks, and the load distribution of data and a corresponding node, the Map to a sub task and stores the task execution node lightest load, and at the request of the job results, assign the corresponding TaskTracker Reduce perform the task;

Wherein the calculation is: MapReduce framework automatically be divided into a set of test parts M, and the data format «id, <A, B>), id represents the log number; B represents a user is currently accessing page; A represents a user access B stay before the page;

Map then input operation for each record scan, the data set is initialized according to the above format; Map after the operation, to obtain an intermediate result << A, B>, 1>, i.e., the user visited the page B from page A; Reduce operation will follow the same intermediate result <a, B> page access mode jumps are combined to obtain an output result << a, B>, n>, where, η behalf of the access path A-> B, a frequency;

Each data set are converted to results list structure Reduce operation, the head of the linked list to save the value of k chain structure: k (A, B) (B, D) (D, E) ..., where, k representing chromosome length; a, B, C, D, E representing the page.

[0016] for each of said internal operation set genetic data, randomly selecting parent chromosomes two chromosomes, and then generates a random insertion position lns, Del erasure position, indels length Len; Comparative isometric whether two chromosomes If it is equal, it is determined whether there is an overlap end to end, if coincident, generating new chromosomes connection, otherwise, not generated offspring chromosomes; if unequal, the determined insertion and deletion of gene segments 2 are the same, if the same, then merged into a chromosome as a new chromosome, otherwise, does not generate offspring chromosomes.

[0017] When genetic generation is 50 or a multiple thereof, the marriage of data between populations k does not change the operation until the operation is terminated;

③ reading task data: Map subtask is assigned to the node reads the already divided TaskTracker good data as input, after the process of generating the key / value pairs;

④Map task execution: TaskTracker Map function call user-written to obtain from JobTracker, and intermediate results are cached in memory;

⑤ the local write intermediate result: the intermediate results in the memory reaches a certain threshold value, written to TaskTracker local disk; process: the intermediate results in the memory reaches a certain threshold value, written to TaskTracker local disk, these intermediate by partitioning the data into R partition function, and transmits the position information thereof to JobTracker local disk, and then sends the location information to JobTracker TaskTracker Reduce perform subtask;

⑥ remote file reading JASReduce the intermediate obtained from TaskTracker JobTracker the subtasks, sorted according to the key value of the intermediate result of the position information by pulling data socket, and the use of intermediate results, with the same key to be combined;

Reduce perform tasks ⑦: Reduce perform tasks TaskTracker traversing the intermediate data of any sort, delivered to the user's Reduce function performed Reduce process;

⑧ output: When all of the Map tasks and Reduce tasks are completed, JobTracker control will result Reduce wrote on HDFS.

Claims (8)

A large-scale web-based data mining Hadoop, characterized in that: the data mining environment structures: and selected to serve as NameNode MapReduce JobTracker servers in the server cluster, to calculate the remaining nodes and the data storage node, the test data set from Web server log server room; data mining job submission: a user submits a job written in a specification based MapReduce programming; assign tasks: calculate the number of tasks Map and Reduce the number of required tasks and the task to a sub Map TaskTracker execution node; while assigning Reduce perform tasks corresponding TaskTracker; read task data: Map subtask is assigned to the node reads the already divided TaskTracker good data as input, after the process of generating the key / value pairs; Map task execution: TaskTracker obtained from call to JobTracker Map user-written functions, and the intermediate results are cached in memory; local write intermediate result: intermediate results in memory reaches a certain threshold, TaskTracker written to the local disk; remote reading of intermediate files JASReduce TaskTracker obtained from the JobTracker subtasks, according to Position information between the result of pulling data through the socket, and the use of intermediate results of key values ​​are sorted, having the same key for merging; perform Reduce Tasks: intermediate data TaskTracker performed Reduce task traverse of any sort, delivered to the user the Reduce function is performed Reduce process; output: when all of the Map tasks and Reduce tasks are completed, JobTracker control will result Reduce wrote on HDFS.

According to claim 1, one of the large-scale web-based data mining Hadoop, wherein said process is assigned tasks: JobTracker job control node according to the situation of the job, the calculated number of tasks required Map and Reduce Tasks number, and the load distribution of data and a corresponding node, the Map to a sub task and stores the task execution node lightest load, and at the request of the job results, assign the corresponding TaskTracker Reduce perform tasks.

According to one of the claims 2 Hadoop massive web-based data mining, characterized in that the local write an intermediate result of the process: the intermediate results in the memory reaches a certain threshold value, is written to the local TaskTracker disk, these intermediate data divided by the partitioning function Reduce the number of tasks in the same partition, and transmits the position information thereof to JobTracker local disk, and then sends the location information to JobTracker Reduce perform subtask TaskTracker0

According to one of the claims 2 massive web-based data mining Hadoop, wherein the task assignment calculation is: MapReduce framework automatically be divided into parts M test set, and the data format «id, <a, B>), id represents the log number; B represents a user is currently accessing page; a represents a page before accessing the user B to stay; Map is then input operation for each record scan data set initialized according to the above format; Map after the operation, to obtain an intermediate result << a, B>, 1>, i.e., the user visited the page B from page a; Reduce operation will follow the same intermediate result <a, B> page jump access method are combined to obtain an output result << a, B>, n>, where, η representatives access path a -> Β frequency.

5. According to one 4 massive claim Hadoop web based data mining, characterized in that each data set, respectively, to convert the result to Reduce operation list structure, the head of the linked list to save the value of k chain structure: k ( a, B) (B, D) (D, E) ..., wherein k represents the length of the chromosome; a, B, C, D, E representing the page.

6. According to one of claim 5 mass Hadoop web-based data mining, characterized in that the interior of each of said sets of genetic data operation, until the k does not change, the operation ends.

The one of the large-scale web of claim 6 data mining methods based on Hadoop, wherein the genetic operation process is: parent chromosome are randomly selected chromosomes, and then randomly generates lns inserted position, delete Del position, indels length Len; Compares two chromosomes length, if they are equal, it is determined whether there is an overlap end to end, if coincident, generating new chromosomes connection, otherwise, not generated offspring chromosomes; if unequal , it is determined that the insertion and deletion of paragraph 2 genes are the same, if the same, then merged into a chromosome as a new chromosome, otherwise, does not generate offspring chromosomes.

8. The member according to claim 7, wherein said web mass data mining methods based on Hadoop, wherein said genetic algebra or a multiple of 50, operation data between populations marriage.