CN106484818A - A kind of hierarchy clustering method based on Hadoop and HBase - Google Patents

Abstract

The invention discloses a hierarchical clustering method based on Hadoop and HBase. This method uses Hadoop to calculate the distance matrix, and converts the result into an HFile file and imports it into HBase with the Bulk Load method. HBase is used to store the distance matrix. It is mainly divided into two tables, one of which is sorted by cluster ID, and the other A table is sorted according to the distance between clusters, which can facilitate the merger of the two closest clusters in each iteration. Finally, implement a multi-threaded algorithm and combine the cache technology to uniformly process the distance matrix in HBase, implement the hierarchical clustering algorithm and reserve multiple adjustable parameters, and the algorithm supports single-linkage, complete-linkage and average-linkage at the same time a clustering method. The scheme proposed by the invention utilizes the parallel computing capability of Hadoop and the massive data storage capability of HBase, thereby improving the large data processing capability and expansibility of the hierarchical clustering algorithm.

Description

A Hierarchical Clustering Method Based on Hadoop and HBase

technical field

The invention relates to the technical field of hierarchical clustering algorithm, Hadoop and HBase, in particular to the design and realization of a hierarchical clustering method based on Hadoop and HBase.

Background technique

As a simple and widely accepted clustering algorithm, hierarchical clustering algorithm has been applied in many fields, such as information retrieval and bioinformatics. The advantage of the hierarchical clustering algorithm is that it expresses the clustering results in a more detailed way. It organizes the clustering relationship between clusters into a dendrogram, and users can clearly know how each cluster is clustered together. Many other clustering algorithms do not give such results. Moreover, compared with other clustering algorithms such as k-means, the hierarchical clustering algorithm does not require the user to specify the number of clusters in advance. Although the hierarchical clustering algorithm has many advantages and is widely accepted and used, with the rapid growth of the data volume, the performance of the single-machine hierarchical clustering algorithm can no longer meet the requirements. The high complexity and inherent limitations of the hierarchical clustering algorithm Data dependencies make it difficult to perform efficiently on large datasets. However, more useful information can only be extracted from more sufficient data, and the size of the data set has become a very important factor in machine learning algorithms. This prompts us to urgently need a hierarchical clustering algorithm that can run on large data sets.

Hadoop is a software framework for distributed processing of massive data, which provides a reliable, efficient, and scalable way to process data. HBase is another important member of the Hadoop ecosystem and is a non-relational distributed database. It provides a highly reliable, high-performance, scalable, column-oriented storage system suitable for unstructured data storage. As very important big data processing technologies, both Hadoop and HBase have been widely used in many big data fields.

The hierarchical clustering algorithm relies on the distance matrix with a space complexity of O(n ² ), which makes it unable to handle large data sets well on a single machine, and also cannot be expanded relatively well. Utilizing Hadoop's parallel computing capability and HBase's high-performance mass data storage capability, it can provide an effective solution for hierarchical clustering algorithms. But currently there is no hierarchical clustering algorithm that can support single-linkage, complete-linkage and average-linkage multiple clustering methods on Hadoop and HBase platforms.

Contents of the invention

The object of the present invention overcomes the above-mentioned deficiency that prior art exists, provides a kind of hierarchical clustering method based on Hadoop and HBase, and supports single-linkage, complete-linkage and average-linkage three kinds of clustering methods simultaneously, specific technical scheme is as follows.

A hierarchical clustering method based on Hadoop and HBase, which uses Hadoop to implement a distance matrix parallel calculation, and uses HBase to store the distance matrix. In the design of the table, the RowKey design is used to make full use of the sorting function of HBase, combined with multi-threading Technology and cache technology to process the distance matrix in HBase, so as to realize a scalable hierarchical clustering method that can be applied to big data.

Further, the parallel calculation of the distance matrix specifically includes: using Hadoop to implement, mainly divided into two MapReduce algorithms, the first MapReduce algorithm calculates the distance and stores the result in an intermediate file, and the second MapReduce algorithm converts the intermediate file result into HFile format; finally use the Bulk Load method to import the result into HBase.

Furthermore, there are mainly two tables in HBase. The RowKey of one of the tables is a Cluster ID pair, and the cluster ID is filled with leading 0s to keep the same length, so that the records in the table can be sorted by the Cluster ID, and the value is The distance between Clusters; quickly obtain the distance related to the specified Cluster ID through this table; the RowKey of the other table is the distance between Clusters, and the leading 0 is added in front of the distance, so that the records in the table can be pressed The distance is sorted from small to large, and the value is a Cluster ID pair; by obtaining the first row of this table, you can quickly obtain the two closest Clusters;

In the initial stage of the clustering method, the two tables are pre-partitioned according to the Cluster ID and the distance in advance, and the partitions are allocated to each node of the HBase cluster.

Further, in the process of clustering, record the merging path of the Cluster and the number of all atomic Clusters contained in the Cluster in the memory; when using the single-linkage or complete-linkage algorithm, calculate the newly synthesized Cluster and the existing Cluster The distance between only needs to retrieve the distance between the two Clusters that synthesized the new Cluster and other Clusters from HBase or cache for new calculation; when using the average-linkage algorithm, in addition to the two Clusters that need to synthesize the new Cluster and other For the distance between Clusters, it is also necessary to obtain the number of all atomic Clusters contained in each Cluster from the memory to calculate the average value.

Further, combined with the multi-threading algorithm of the cache, the two closest Clusters are obtained from HBase and merged into a new Cluster, and the distance between the two closest Clusters and other Clusters is obtained in parallel from the cache or HBase During this period, the deletion thread is started to delete the invalid data in HBase, and the calculation thread is started to calculate the distance between the cluster that has obtained the distance information and the new Cluster in advance, and at the same time write the new distance information back to the cache or HBase in parallel; here, the cache technology is used to Reduce the network IO of the algorithm.

Furthermore, Hadoop is used to calculate the distance matrix of the algorithm, HBase is used to store the distance matrix, and the algorithm is designed and implemented by combining multi-threading technology and cache technology, thereby improving the scalability of the hierarchical clustering algorithm and the ability to process large data.

Compared with the prior art, the present invention has the following advantages and technical effects:

The present invention implements a distance matrix parallelization calculation algorithm based on Hadoop, and converts the result into an HFile file and imports it into HBase with the Bulk Load method. HBase is used to store the distance matrix. In the design of the table, the unique RowKey design is used to make full use of the sorting function of HBase, which is convenient for the algorithm to quickly obtain the distance information and obtain the two closest clusters. At the same time, the algorithm is designed and implemented by combining multithreading and cache technology, and a number of adjustable parameters are reserved. Through Hadoop's parallel computing capability and HBase's massive data storage capability, the scalability and big data processing capability of the hierarchical clustering algorithm are improved.

Description of drawings

Figure 1 is the architecture diagram of the distance matrix calculation algorithm in the example.

Figure 2 is a schematic diagram of the table design in the example.

Fig. 3 is a schematic diagram of the hierarchical tree of the hierarchical clustering algorithm in the example.

FIG. 4 is a schematic diagram of thread execution sequence.

Fig. 5 is an explanatory diagram of algorithm parameters in the example.

detailed description

In order to make the technical solutions and advantages of the present invention clearer, further detailed description will be given below in conjunction with the accompanying drawings, but the implementation and protection of the present invention are not limited thereto. It should be pointed out that, if there are symbols or processes in the following that are not specifically described in detail, those skilled in the art can understand or implement them with reference to the prior art.

1. Parallel calculation algorithm of distance matrix

The parallel calculation algorithm of the distance matrix aims to improve the calculation speed of the distance matrix and quickly import it into HBase. In the process of clustering, the hierarchical clustering algorithm needs to rely on a distance matrix with a space complexity of O(n ² ). In this method, a Hadoop-based parallel computing algorithm is designed and implemented for the calculation of the distance matrix. As shown in Figure 1. The algorithm first distributes the file containing the data to be clustered as a global cache file to each task, and then divides the file containing the ClusterID into blocks, and each task processes one block. Each task iteration obtains each ID, and calculates the distance between it and all other Clusters whose ID value is greater than it, such as the Cluster with ID = 2, calculates the distance between it and the Cluster with ID = 3, 4, 5... distance, and output the result to an intermediate file in the format of (distance ID1, ID2timestamp). For a distance matrix with a space complexity of O(n ² ), if you insert data into HBase one by one in reduce, the speed will be relatively slow. In this implementation method, the Bulk Load method is used to quickly import data into HBase. The Bulk Load method is based on the principle that HBase data is stored on HDFS in a specific format. Therefore, in this implementation method, another MapReduce program is implemented to convert the intermediate result into a HFile format file, and then the Bulk Load method is used to import the HFile into HBase. Using Hadoop and Bulk Load methods, the parallel calculation of distance matrix and fast import to HBase are realized.

2. Table design

The design of the table on HBase has a great impact on the performance of the algorithm. An unreasonable design will greatly reduce the performance of the algorithm and affect the usability of the algorithm. In this implementation, the table is designed uniquely according to the characteristics of the hierarchical clustering algorithm, as shown in Figure 2. There are two main tables in HBase: distanceMatrix table and sortedDistance table. The distanceMatrix table stores the distance matrix data sorted from small to large according to the Cluster ID pair. Its RowKey is the Cluster ID pair, and the Cluster ID is filled with leading 0 to keep the RowKey length consistent, so that the records in the table can be sorted according to the Cluster ID, value is the distance between Clusters. Through this table, you can quickly obtain the distance related to the specified Cluster ID. The sortedDistance table stores distance matrix data sorted by distance from small to large. Its RowKey is the distance between Clusters, and a leading 0 is added in front of the distance to keep all RowKeys of the same length, so that the records in the table can be sorted from small to large distances. Sort, value is a Cluster ID pair. By obtaining the first row of this table, you can quickly obtain the two closest Clusters. At the same time, the implementation reserves parameters to specify the initial number of regions of the two tables. In the initial stage of the algorithm, the two tables are pre-partitioned according to the Cluster ID and distance, and the partitioned regions are allocated to each node of the HBase cluster. On the one hand, improve the concurrent processing capability of HBase.

3. Simplify the calculation

When calculating the distance between two clusters, if the two clusters are synthesized by other clusters, we need to use the distance of all point pairs between the two clusters, such as singleton cluster A and singleton cluster B synthesis If cluster D is selected, the distance between cluster C and singleton cluster A and the distance between cluster C and singleton cluster B are used to calculate the distance between singleton cluster C and cluster D. But in fact, it is not necessary to compare the distances of all singleton clusters every time, but only need to compare the distances between the sub clusters under the two clusters. Figure 3 is a hierarchical tree of the agglomerative hierarchical clustering algorithm. We define dis(A,B) as the distance between cluster A and cluster B, min as the minimum value, max as the maximum value, avg as the average value, and count(A) as the number of all points contained in cluster A number. When using the single-linkage method to calculate the distance dis(7,9) between cluster 7 and cluster 9:

dis(7,9)=min(dis(1,5),dis(1,6),dis(2,5),dis(2,6),dis(3,5),dis(3,6) )

＝min(min(dis(1,5),dis(1,6)),min(dis(2,5),dis(2,6)),min(dis(3,5),dis(3, 6)))

＝min(dis(1,7),dis(2,7),dis(3,7))

＝min(dis(1,7),min(dis(2,7),dis(3,7)))

＝min(dis(1,7),dis(8,7))

It can be seen that when calculating the distance dis(7,9) between cluster 7 and cluster 9, only dis(1,7) and dis(8,7) are needed, instead of all the distances between cluster1-4 and cluster4-5. The complete-linkage method is similar to the single-linkage method, only need to replace the min in the above formula with max. When using the average-linkage method to calculate the distance dis(7,9) between cluster 7 and cluster 9:

It can be seen that when using average-linkage, not only dis(1,7) and dis(8,7), but also the number of points contained in cluster7 and cluster9 are required, so that the average value can be calculated.

4. Design and implementation of multi-threaded algorithm combined with cache

In addition to an efficient access method for the distance matrix, a parallel algorithm is needed to complete the clustering process. In the algorithm, we combine multi-thread technology and cache technology to complete the design and implementation of the algorithm. According to the principle of hierarchical clustering algorithm, it is first necessary to obtain the two closest clusters and merge them into a new cluster. As mentioned above, the sortedDistance table stores cluster pairs and the distance between them. The most important thing is that they are sorted by distance from small to large, so we only need to get the first record of the sortedDistance table. We use HBase's scan API to get the first piece of data from the sortedDistance table. In order to avoid getting redundant data, we set the scan cache to 1.

For the convenience of illustration, assume that the initial data set has 10 points, each point is regarded as a cluster at the beginning, and there are 10 clusters at the beginning: C1 to C10. Suppose the two nearest clusters are C1 and C2 in the first iteration, and we merge them into a new cluster C11. According to the previous section, we need the distance between C1, C2 and cluster C3 to C10 to calculate C11 and C3...C10, and write the new distance to cache or HBase. In the case of missing the cache, we need to read the distance from HBase, because this is an IO-intensive operation, so we use a multi-threaded method to concurrently use the Scan API to read the relevant distance from the distanceMatric table, At the same time, we adjust the number of lines that Scan reads back each time through scan.setCaching. In the process of reading the distance from the distanceMatric table, we also start multiple threads in parallel to delete the distances related to the two clusters that have been merged from the sortedDistance table. Obviously, get and put operations are IO-intensive operations, so We can calculate the acquired data in advance without having to wait until all the data has been acquired. So when we start the scan thread to read data in HBase, we also start the calculation thread to perform calculations instead of waiting for the scan thread to end. And we also start multiple threads in parallel to write the new distance back to the cache or HBase, so that when all the data is obtained, the calculation task has basically been completed.

The execution sequence of each parallel thread is shown in FIG. 4 . As we know, we need some synchronization techniques to control threads. In our algorithm, we use BlockingQueue and Barrier technology. BlockingQueue controls thread communication by controlling two threads to alternately put and take elements into BlockingQueue. The Barrier technology is very useful in parallel iterative algorithms. This algorithm splits a problem into multiple sub-problems and executes them in parallel. When it reaches the barrier position, it will wait until all threads have reached the barrier position.

5. Description of adjustable parameters

In the process of using Hadoop and HBase, there are many parameters that need to be adjusted according to the actual application requirements to achieve a better performance. In the design process of our algorithm, in addition to the parameters supported by Hadoop and HBase itself, we also reserved many algorithm-defined parameters for the convenience of adjusting according to the actual situation in the process of using the algorithm. The main parameters are shown in Figure 5 . The main customization parameters are described next.

The distance matrix generated during the calculation of the hierarchical clustering algorithm is relatively large. We put it in the HBase table and add a layer of cache on the client side. In order to make full use of the performance of multiple machines in the HBase cluster, it is better to distribute the load to each machine. Therefore, when building a table, we pre-partition the table, split a table into multiple regions, and then Each server is responsible for a part of the region. We provide regionCountDM and regionCountSD parameters to specify how many regions the distanceMatric table and sortedDistance table are divided into. At the same time, we maintain a layer of cache on the client side to improve the performance of data lookup, and provide the cacheSize parameter to adjust the number of cached records according to the actual situation. Corresponding to the single-linkage, complete-linkage and average-linkage methods of the hierarchical clustering algorithm, we provide the similarity_method parameter to specify which method to use. At the same time, distance_method is also provided to specify which distance measurement method to use to calculate the distance between two points. We only implemented the Euclidean method during the test. In the calculation process of the algorithm, we use multi-threading technology, so we also provide parameters to adjust the number of threads. We can adjust the number of put threads by adjusting the putThreadNum parameter, and at the same time adjust the pagesNum parameter to control the thread that reads data from HBase number, we will page the data that needs to be read from HBase, each thread reads one page, and the pagesNum parameter controls the number of pages, such as pagesNum=10, we need to read data with IDs from 0 to 10000, then divide For 10 threads, read 0 to 1000, 1001 to 2000 and so on. Finally, in order to control the end condition of the hierarchical clustering algorithm, you can set the maxClusterNum parameter to specify how many clusters the data will be clustered into, or you can set the minDistance parameter to specify when the minimum distance between two clusters is less than the value specified by minDistance End clustering.

Claims (5)

1. A hierarchical clustering method based on Hadoop and HBase, characterized in that Hadoop is used to realize a distance matrix parallelization calculation, and HBase is used to store the distance matrix, and the RowKey design is adopted in the design of the table to make full use of the sorting function of HBase, Combined with multi-threading technology and cache technology to process the distance matrix in HBase, a scalable hierarchical clustering method that can be applied to big data is realized. 2. the hierarchical clustering method based on Hadoop and HBase according to claim 1, it is characterized in that the parallelization calculation of distance matrix specifically comprises: adopt Hadoop to realize, mainly be divided into two MapReduce algorithms, the first MapReduce algorithm calculates distance And store the result in the intermediate file, the second MapReduce algorithm converts the intermediate file result into HFile format; finally use the Bulk Load method to import the result into HBase. 3. the hierarchical clustering method based on Hadoop and HBase according to claim 1 is characterized in that there are mainly two tables in HBase, wherein the RowKey of a table is a Cluster ID pair, and the Cluster ID is completed leading 0 and kept The same length, so that the records in the table can be sorted by Cluster ID, and the value is the distance between Clusters; through this table, the distance related to the specified Cluster ID can be quickly obtained; the RowKey of the other table is the distance between Clusters , and complete the leading 0 in front of the distance, so that the records in the table can be sorted according to the distance from small to large, and the value is a ClusterID pair; by obtaining the first row of this table, you can quickly obtain the two closest Clusters ; In the initial stage of the clustering method, the two tables are pre-partitioned according to the Cluster ID and the distance in advance, and the partitions are allocated to each node of the HBase cluster. 4. the hierarchical clustering method based on Hadoop and HBase according to claim 1, is characterized in that: in the process of clustering, the merge path of Cluster and the quantity of all atomic Clusters that Cluster comprises are recorded in internal memory; When the single-linkage or complete-linkage algorithm is used, to calculate the distance between the newly synthesized Cluster and the existing Cluster, you only need to retrieve the distance between the two Clusters that synthesized the new Cluster and the remaining Clusters from HBase or cache for new calculation. Calculation; when the average-linkage algorithm is used, in addition to the distance between the two clusters of the new cluster and other clusters, it is also necessary to obtain the number of all atomic clusters contained in each cluster from the memory to calculate the average value. 5. the hierarchical clustering method based on Hadoop and HBase according to claim 1, is characterized in that in conjunction with the multithreading algorithm of cache, obtains two Clusters closest to distance from HBase and merges them into a new Cluster, and parallel Acquire the distance information between the two closest Clusters and other Clusters from the cache or HBase. During this period, start the deletion thread to delete the invalid data in HBase and start the calculation thread to calculate the distance between the Cluster that has obtained the distance information and the new Cluster in advance. At the same time, write the new distance information back to the cache or HBase in parallel.