CN103218404B - A kind of multi-dimensional metadata management method based on associate feature and system - Google Patents

Abstract

The invention discloses a multi-dimensional metadata management method based on association characteristics, which includes: in the metadata server cluster, dividing the metadata on each metadata server according to the association characteristics, so as to generate metadata collections and collection statistics files , according to the collection statistics file, the metadata cluster is grouped to generate multiple metadata server groups and group configuration files; according to the collection statistics file, a local index table is established on each metadata server, and according to the group Collect statistical files, establish a group index table in each metadata server group, build the top-level index table of the metadata server cluster according to the group index table, receive query requests from users, and query the top-level index table, group Index tables and local index tables. The invention can make full use of the association characteristics among the multi-dimensional attributes of the metadata, meet the requirement of complex query, and has good scalability.

Description

A method and system for managing multi-dimensional metadata based on association characteristics

technical field

The invention belongs to the field of computer data storage, and more specifically relates to a multi-dimensional metadata management method and system based on association characteristics.

Background technique

With the advent of the era of cloud computing and cloud storage, the geometric progression of the data scale in the information storage system makes efficient storage, management and query of data more and more difficult. The continuous growth of massive data scale has led to the increasing difficulty of data storage and maintenance. Studies have shown that the file data of massive storage systems in practice has significant correlation characteristics. Correlation feature refers to the aggregation phenomenon of files in their attribute space, which essentially reflects the correlation between files. Usually, we often use temporal correlation and spatial correlation between files. Time correlation means that files with similar time will be accessed immediately within a period of time, while spatial correlation means that files located in adjacent locations It is highly likely to be accessed by subsequent requests. In addition to temporal correlation and spatial correlation, there are many correlations between files, such as file size, file access frequency, file creator, and so on. However, the existing research results obviously lack the research on the relevance of files in more attributes. Considering the correlation of more attributes helps to distinguish the correlation between files more accurately. Based on the distance measure in the multidimensional attribute space, the correlation between two files can be clearly calculated. In the face of massive data processing, using certain methods to measure the correlation between data, and thus divide the data into multiple aggregated spaces, will bring obvious benefits to subsequent processing.

However, existing metadata management methods have the following problems:

(1) The correlation characteristics between the multi-dimensional attributes of metadata are not fully utilized. The existing methods often only use the time attributes and spatial attributes of metadata, and do not fully exploit the correlation characteristics between metadata.

(2) It cannot effectively support complex query requests. Existing methods cannot effectively handle query requests involving multi-dimensional attributes of metadata, such as range queries and TopK queries;

(3) The scalability is poor. When the number of metadata increases with the expansion of the system, the query response time of the existing method will increase significantly.

Contents of the invention

Aiming at the defects of the prior art, the purpose of the present invention is to provide a multi-dimensional metadata management method based on association characteristics, aiming at solving the problem of metadata management in mass storage systems, which can make full use of the relationship between the multi-dimensional attributes of metadata Correlation features meet complex query requirements and have good scalability.

In order to achieve the above object, the present invention provides a multi-dimensional metadata management method based on association characteristics, comprising the following steps:

(1) In the metadata server cluster, the metadata on each metadata server is divided according to the associated characteristics to generate metadata collections and collection statistics files;

(2) According to the collection statistics file, the metadata cluster is grouped to generate multiple metadata server groups and group configuration files;

(3) According to the collection statistics file, establish a local index table on each metadata server; the local index table is used to manage the metadata collection on each metadata server, and each item in the index table records the data in the collection statistics file. The metadata collection number of the metadata collection number, and the storage address of the metadata collection corresponding to the metadata collection number in the disk;

(4) according to the group configuration file and the collection statistics file, set up a group index table in each metadata server group respectively;

(5) according to the group index table, establish the top-level index table of the metadata server cluster;

(6) Receive the query request from the user, and query the top-level index table, group index table and local index table sequentially according to the query request, and return the query result; wherein the user query request includes point query, range query and TopK query.

Step (1) includes the following sub-steps:

(1-1) Determining the multi-dimensional attributes representing the associated characteristics between metadata on each metadata server;

(1-2) Constructing the multi-dimensional attribute of metadata into a fixed-length input vector, which is used as the input value of the position-sensitive hash function;

(1-3) Use the same position-sensitive hash function to perform hash calculation on the input vector, and the obtained hash value is used as the unique identifier of the metadata corresponding to the input vector;

(1-4) Divide the metadata with the same hash value into the same metadata set, and use the hash value as the number of the metadata set;

(1-5) Statistics on the division of metadata in the metadata collection to generate a collection statistics file; the collection statistics file includes the number of the metadata collection, the number of metadata, the average value of each dimension attribute, and the range of each dimension attribute, among which the metadata The set numbers range from 1, 2, 3, ..., N, where N represents the length of the hash table in the position-sensitive hash function.

Step (2) is specifically to construct a bit vector on each metadata server, the length of the bit vector is the same as the length of the hash table used by the location-sensitive hash function in step (1), and then, according to all metadata The Hamming distance between the bit vectors of the servers and the hierarchical clustering algorithm are used to perform clustering operations between the metadata servers to obtain the grouping of the metadata servers. When the number of groups formed by clustering reaches the lower limit, or the grouping If the distance between them reaches the upper limit, the clustering operation will be stopped to obtain multiple metadata server groups, and the results will be saved in the grouping configuration file.

Step (4) is specifically, for each group in the group configuration file, construct a corresponding group index table, and each item in the group index table records the information of metadata sets on all metadata servers in the group, including metadata Data collection number, IP address of the metadata server where the metadata collection is located, number of metadata, average value of attributes of each dimension, and range of attributes of each dimension.

The point query operation in step (6) specifically includes the following steps:

(6-1-1) Receive a point query request, determine the multidimensional attribute of the metadata corresponding to the query request, and use the location-sensitive hash function to calculate the hash value of the multidimensional attribute, which is the metadata to be queried the number of the collection;

(6-1-2) Query the entry corresponding to the metadata collection number in the top-level index table, to obtain the IP address of the metadata server group where the metadata is located;

(6-1-3) Determine the corresponding metadata server according to the IP address of the metadata server group, and query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata where the metadata is located. IP address of the data server;

(6-1-4) According to the found IP address of the metadata server, query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata set where the metadata is located in the disk storage address;

(6-1-5) According to the storage address of the found metadata set in the disk, query the corresponding metadata set and return the query result.

The range query operation in step (6) specifically includes the following steps:

(6-2-1) Receive a range query request, determine the multi-dimensional attribute range to be checked, calculate the median value of each attribute range, and construct an input vector from the median value of each attribute range, and use the position-sensitive hash function to calculate the input The hash value of the vector, which is the number of the metadata set to be queried;

(6-2-2) Query the table item corresponding to the metadata set number in the top-level index table, compare the multi-dimensional attribute range in the query request with the multi-dimensional attribute range saved in the table item, if the two ranges do not intersect, directly The returned result is empty; if the two ranges intersect, the IP address of the metadata server group containing the metadata to be checked is obtained;

(6-2-3) Determine the corresponding metadata server according to the IP address of the metadata server group, and query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata where the metadata is located. IP address of the data server;

(6-2-4) According to the found IP address of the metadata server, query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata set where the metadata is located in the disk storage address;

(6-2-5) According to the storage address of the found metadata set in the disk, query the corresponding metadata set, and return all the metadata that meet the multi-dimensional attribute range in the query request.

The TopK query operation in step (6) specifically includes the following steps:

(6-3-1) Receive the TopK query request, determine the multi-dimensional attribute and K value of the metadata corresponding to the query request, and use the location-sensitive hash function to calculate the hash value of the multi-dimensional attribute, which is the required query The number of the metadata collection; where K represents the number of metadata that is closest to the multidimensional attribute of the metadata in the query request;

(6-3-2) Query the table item corresponding to the metadata collection number in the top-level index table. If the number of metadata recorded in the table item is less than the K value, the table items on the left and right sides of the table item are also included in the query range. Until the sum of the metadata numbers in the entry is greater than or equal to the K value, finally obtain the IP addresses of multiple metadata server groups;

(6-3-3) Determine the corresponding metadata server according to the IP address of the metadata server group, and query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata where the metadata is located. IP address of the data server;

(6-3-4) According to the found IP address of the metadata server, query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata set where the metadata is located in the disk storage address;

(6-3-5) Query the corresponding metadata set according to the storage address of the found metadata set in the disk, and return K pieces of metadata that are most similar to the multidimensional attributes of the metadata in the query request.

Through the above technical solutions conceived by the present invention, compared with the prior art, this method has the following beneficial effects:

1. Make full use of the association characteristics between multi-dimensional attributes of metadata. Due to the adoption of steps (1) and (2), metadata are divided into multiple metadata sets according to the correlation characteristics between their multi-dimensional attributes, and metadata with the same or similar multi-dimensional attributes are divided into the same metadata set. , so that all metadata can be effectively managed in units of metadata collections.

2. Effectively support complex query requests, such as range query and TopK query. Due to the adoption of step (3), step (4) and step (5), for each query request, the top-level index table, the group index table and the local index table will be queried in turn, and finally the query will be located among the metadata collections middle. Query the metadata with the same or similar multi-dimensional attributes in the metadata collection, and the results can be returned quickly and accurately.

3. Meet the scalability requirements. Because the metadata is divided into multiple metadata collections for management according to the associated characteristics, the rapid growth of the system metadata will only cause the slow growth of the metadata collection, thus ensuring the effectiveness and efficiency of metadata management.

Another object of the present invention is to provide a multi-dimensional metadata management system based on association characteristics, aiming at solving the problem of metadata management in mass storage systems, which can make full use of the association characteristics between multi-dimensional attributes of metadata to meet complex Query requirements and have good scalability.

In order to achieve the above object, the present invention provides a multi-dimensional metadata management system based on association characteristics, including a metadata set generation module, a metadata server grouping module, a local index generation module, a group index generation module, a top-level index generation module, a query Module, the metadata collection generation module divides the metadata on each metadata server according to the associated characteristics to generate metadata collections and collection statistics files, and the metadata server grouping module performs grouping operations on metadata clusters according to the collection statistics files , to generate multiple metadata server groups and group configuration files, the local index generation module builds local index tables on each metadata server according to the collection statistics file, and the group index generation module according to the group configuration files and collection statistics files, Create a group index table in each metadata server group, the top-level index generation module builds the top-level index table of the metadata server cluster according to the group index table, and the query module receives query requests from users, and queries the top-level indexes in turn according to the query requests table, group index table, and local index table, and return query results.

The metadata set generation module includes a multidimensional attribute determination module, an input vector construction module, a hash function calculation module, an association characteristic division module and an output module. The multidimensional attribute determination module determines the multidimensional attributes representing the association characteristics between metadata, and the input vector construction module Construct the multi-dimensional attributes of metadata into a fixed-length input vector, which is used as the input value of the position-sensitive hash function, and the hash function calculation module uses the same position-sensitive hash function to perform hash calculation on the input vector, and the obtained The hash value is used as the unique identifier of the metadata corresponding to the input vector, and the associated feature division module divides the metadata with the same hash value into the same metadata set, and uses the hash value as the number of the metadata set, The output module counts the division of metadata in the metadata collection to generate collection statistics files.

The query module specifically includes a point query sub-module, a range query sub-module, and a TopK query sub-module. The point query sub-module processes the user's point query request, and given the multi-dimensional attributes of a certain metadata, the query result returns the specific information of the metadata, the range The query sub-module processes the user's range query request, given the range of multi-dimensional attributes, the query result returns all metadata information in the entire system that meets the range, the TopK query sub-module processes the user's TopK query request, given a set of multi-dimensional attributes, and Specify the K value, and the query result returns the K pieces of data that are closest to the given multidimensional attribute in the entire system.

Through the above technical solutions conceived by the present invention, compared with the prior art, this system has the following beneficial effects:

1. Make full use of the association characteristics between multi-dimensional attributes of metadata. Due to the adoption of the metadata set generation module and the metadata server grouping module, metadata is divided into multiple metadata sets according to the correlation characteristics between their multidimensional attributes, and metadata with the same or similar multidimensional attributes are divided into the same metadata In the collection, all metadata can be effectively managed in units of metadata collections.

2. Effectively support complex query requests, such as range query and TopK query. Due to the use of the local index generation module, group index generation module, and top-level index generation module, for each query request, the top-level index table, group index table, and local index table will be queried in turn, and finally the query will be located among the metadata collections. middle. Query the metadata with the same or similar multi-dimensional attributes in the metadata collection, and the results can be returned quickly and accurately.

3. Meet the scalability requirements. Because the metadata is divided into multiple metadata collections for management according to the associated characteristics, the rapid growth of the system metadata will only cause the slow growth of the metadata collection, thus ensuring the effectiveness and efficiency of metadata management.

Description of drawings

FIG. 1 is a flow chart of the multi-dimensional metadata management method based on association characteristics in the present invention.

Fig. 2 is a block diagram of the multi-dimensional metadata management system based on the association feature of the present invention.

Fig. 3 is a detailed flowchart of step (1) in the method of the present invention.

Fig. 4 is a schematic block diagram of a metadata set generating module of the present invention.

Fig. 5 is a schematic diagram of the query process of the present invention.

FIG. 6 is a block diagram of the query module in the system of the present invention.

Fig. 7 is a flow chart of the point query process of the present invention.

Fig. 8 is a flow chart of the scope query process of the present invention.

Fig. 9 is a flow chart of the TopK query process of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The present invention is a multi-dimensional metadata management method and system based on association characteristics. The method uses the association characteristics between multi-dimensional metadata to perform grouping operations on metadata, and divides metadata with the same or similar multi-dimensional attributes into the same metadata set. Among them, various complex query operations on metadata can be satisfied.

As shown in Figure 1, the multi-dimensional metadata management method based on association characteristics of the present invention includes the following steps:

(1) In the metadata server cluster, the metadata on each metadata server is divided according to the associated characteristics to generate metadata collections and collection statistics files;

As shown in Figure 3, this step includes the following sub-steps,

(1-1) Determine the multi-dimensional attribute representing the correlation characteristics between metadata on each metadata server; in this embodiment, the multi-dimensional attribute includes file size, file creation time, file modification time, file access times, etc.;

(1-2) Construct the multi-dimensional attributes of metadata into a fixed-length input vector, which is used as the input value of the position-sensitive hash function; specifically, the size of the fixed length is equal to the dimension of the metadata; the constructed input Vectors are represented as (file size, file creation time, file modification time, number of file accesses, ...);

(1-3) Use the same position-sensitive hash function to perform hash calculation on the input vector, and the obtained hash value is used as the unique identifier of the metadata corresponding to the input vector;

(1-4) Divide the metadata with the same hash value into the same metadata set, and use the hash value as the number of the metadata set;

(1-5) Statistics on the division of metadata in the metadata collection to generate a collection statistics file; the collection statistics file includes the number of the metadata collection, the number of metadata, the average value of each dimension attribute, and the range of each dimension attribute, among which the metadata The set numbers range from 1, 2, 3, ..., N, where N represents the length of the hash table in the position-sensitive hash function.

There are many ways to group metadata based on association properties, and the present invention chooses position-sensitive hashing. Position-sensitive hashing can map multidimensional data into one-dimensional space while keeping the spatial relationship between multidimensional data unchanged, that is, originally adjacent data is still adjacent after being hashed. For each piece of metadata, the hash value is calculated by position-sensitive hashing, and the metadata with the same hash value is aggregated into the same metadata set, so that the purpose of grouping metadata can be achieved.

The advantage of this step is that all the metadata is divided into multiple metadata sets by using the correlation feature between the multi-dimensional attributes of the metadata, and the metadata contained in the same metadata set have the same or similar multi-dimensional attributes. In this way, the query operation for metadata can be limited to the corresponding metadata collection, thereby significantly improving query efficiency.

(2) According to the collection statistics file, the metadata cluster is grouped to generate multiple metadata server groups and group configuration files;

Specifically, a bit vector is constructed on each metadata server, the length of the bit vector is the same as the length of the hash table used by the position-sensitive hash function in step (1), and the first bit of the bit vector corresponds to the metadata Whether the metadata set whose data set number is 1 exists, if it exists, the first bit is 1, otherwise it is 0, ..., the Nth bit of the bit vector corresponds to whether the metadata set whose number is N exists, if it exists, the Nth bit is 1, otherwise it is 0; then, according to the Hamming distance between the bit vectors of all metadata servers and using the hierarchical clustering algorithm in the metadata server Clustering operations are performed to obtain groups of metadata servers. When the number of groups formed by clustering reaches the lower limit, or the distance between groups reaches the upper limit, the clustering operation is stopped to obtain multiple metadata server groups. And save the result in the group configuration file; in this embodiment, the value of the upper limit is equal to half the length of the bit vector, and the value of the lower limit is 1.

For example, if there are 5 metadata servers A, B, C, D, and E, corresponding to 5 bit vectors, first calculate the Hamming distance between any two of these 5 bit vectors, and then select the Haiming distance The metadata servers corresponding to the two bit vectors with the shortest distance (such as A and B) form a cluster F, and each cluster generated is a metadata server group, and then the clustering is performed among F, C, D, and E. The iterative operation is repeated, and once the number of groups formed reaches the lower limit, or the distance between groups reaches the upper limit, the clustering operation is stopped.

The advantage of this step is that dividing the metadata server into multiple metadata server groups can effectively disperse query requests among multiple metadata server groups, thereby avoiding system bottlenecks.

(3) According to the collection statistics file, establish a local index table on each metadata server; the local index table is used to manage the metadata collection on each metadata server, and each item in the index table records the data in the collection statistics file. The metadata collection number of the metadata collection number, and the storage address of the metadata collection corresponding to the metadata collection number in the disk.

(4) According to the group configuration file and the collection statistics file, establish a group index table in each metadata server group respectively; specifically, for each group in the group configuration file, build a corresponding group index table, group index Each item in the table records the information of the metadata collection on all metadata servers in the group, including the metadata collection number, the IP address of the metadata server where the metadata collection is located, the number of metadata, the average value of each dimension attribute, and the attribute range.

For the established group index table, in specific implementation, it can be stored on any metadata server in the metadata server group, and this metadata server is called the group leader metadata server of the metadata server group; In consideration of data redundancy, the group index table can also be stored on multiple metadata servers in the metadata server group.

(5) According to the group index table, establish the top-level index table of the metadata server cluster; specifically, establish the top-level index table to manage the information of the metadata server group, including the metadata collection number, the IP address of the group where the metadata collection is located, The number of metadata, the average value of each dimension attribute, and the range of each dimension attribute.

The top-level index table can be stored on any selected metadata server, or it can be stored on multiple metadata servers to avoid a single point of failure.

Steps (3) to (5) are used to construct a three-level index table for the entire system. Through the three-level index table, the query request for metadata can be quickly located in a metadata server group, and then located in a specific metadata server, and finally find the metadata collection containing the metadata. The three-level index table is used to quickly locate metadata collections and improve query time efficiency.

(6) Receive the query request from the user, and query the top-level index table, group index table and local index table sequentially according to the query request, and return the query result; the user query request specifically includes: point query, range query and TopK query.

As shown in Figure 6, the point query operation in step (6) specifically includes the following steps:

(6-1-1) Receive a point query request, determine the multidimensional attribute of the metadata corresponding to the query request, and use the location-sensitive hash function to calculate the hash value of the multidimensional attribute, which is the metadata to be queried the number of the collection;

(6-1-2) Query the entry corresponding to the metadata collection number in the top-level index table, to obtain the IP address of the metadata server group where the metadata is located;

(6-1-3) Determine the corresponding metadata server according to the IP address of the metadata server group, and query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata where the metadata is located. IP address of the data server;

(6-1-4) According to the found IP address of the metadata server, query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata set where the metadata is located in the disk storage address;

(6-1-5) Query the corresponding metadata set according to the storage address of the found metadata set in the disk, and return the query result;

As shown in Figure 7, the range query operation in step (6) specifically includes the following steps:

(6-2-1) Receive a range query request, determine the multi-dimensional attribute range to be checked, calculate the median value of each attribute range, and construct an input vector from the median value of each attribute range, and use the position-sensitive hash function to calculate the input The hash value of the vector, which is the number of the metadata set to be queried;

(6-2-2) Query the table item corresponding to the metadata set number in the top-level index table, compare the multi-dimensional attribute range in the query request with the multi-dimensional attribute range saved in the table item, if the two ranges do not intersect, directly The returned result is empty; if the two ranges intersect, the IP address of the metadata server group containing the metadata to be checked is obtained;

(6-2-3) Determine the corresponding metadata server according to the IP address of the metadata server group, and query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata where the metadata is located. IP address of the data server;

(6-2-4) According to the found IP address of the metadata server, query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata set where the metadata is located in the disk storage address;

(6-2-5) According to the storage address of the found metadata set in the disk, query the corresponding metadata set, and return all the metadata that meet the multi-dimensional attribute range in the query request;

As shown in Figure 8, the TopK query operation in step (6) specifically includes the following steps:

(6-3-1) Receive the TopK query request, determine the multi-dimensional attribute and K value of the metadata corresponding to the query request, and use the location-sensitive hash function to calculate the hash value of the multi-dimensional attribute, which is the required query The number of the metadata collection; where K represents the number of metadata that is closest to the multidimensional attribute of the metadata in the query request;

(6-3-2) Query the table item corresponding to the metadata collection number in the top-level index table. If the number of metadata recorded in the table item is less than the K value, the table items on the left and right sides of the table item are also included in the query range. Until the sum of the metadata numbers in the entry is greater than or equal to the K value, finally obtain the IP addresses of multiple metadata server groups;

(6-3-3) Determine the corresponding metadata server according to the IP address of the metadata server group, and query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata where the metadata is located. IP address of the data server;

(6-3-4) According to the found IP address of the metadata server, query the entry corresponding to the metadata set number in the group index table of the metadata server to obtain the metadata set where the metadata is located in the disk storage address;

(6-3-5) According to the storage address of the found metadata collection in the disk, query the corresponding metadata collection, and return the K pieces of metadata closest to the multidimensional attributes of the metadata in the query request;

Because metadata is composed of multi-dimensional attributes, these attributes include file size, file creation time, file modification time, file access times, and so on. In the file system, there are often certain correlation characteristics between metadata, such as similar size and access time. The traditional metadata query method only uses the spatial locality and temporal locality of metadata, but ignores metadata. Relationships among other attributes. In the face of complex query requests, such as TopK query and range query, traditional metadata management methods have to traverse the entire metadata collection to obtain results. In the present invention, the correlation characteristics between metadata are fully utilized, and metadata with the same or similar metadata attributes are focused into a metadata collection. For each metadata query request, the present invention will first locate the query to one or some metadata collections, thereby greatly reducing the number of metadata queries and returning query results faster.

As shown in Figure 2, the multi-dimensional metadata management system based on the association feature of the present invention includes a metadata collection generation module 1, a metadata server grouping module 2, a local index generation module 3, a group index generation module 4, a top-level index generation module 5, Query module 6.

Metadata collection generation module 1, divides the metadata on each metadata server according to the associated characteristics to generate metadata collection and collection statistics files; as shown in Figure 4, metadata collection generation module 1 includes a multi-dimensional attribute determination module 11. An input vector construction module 12 , a hash function calculation module 13 , an association characteristic division module 14 and an output module 15 .

The multidimensional attribute determining module 11 determines a multidimensional attribute representing an association characteristic between metadata;

The input vector construction module 12 constructs the multi-dimensional attribute of the metadata into a fixed-length input vector, which is used as the input value of the position-sensitive hash function;

The hash function calculation module 13 uses the same position-sensitive hash function to perform hash calculation on the input vector, and the obtained hash value is used as the unique identifier of the metadata corresponding to the input vector;

The association feature division module 14 divides the metadata with the same hash value into the same metadata set, and uses the hash value as the number of the metadata set;

The output module 15 counts the division of metadata in the metadata collection to generate a collection statistics file;

The metadata server grouping module 2 performs grouping operations on the metadata clusters according to the collection statistics file, so as to generate multiple metadata server groups and group configuration files;

The local index generating module 3, according to the collection statistics file, respectively establishes a local index table on each metadata server;

The group index generation module 4 establishes a group index table in each metadata server group respectively according to the group configuration file and the collection statistics file;

The top-level index generation module 5 establishes the top-level index table of the metadata server cluster according to the group index table;

Query module 6 receives query request from user, and inquires top-level index table, group index table and local index table successively according to query request, and returns query result; As shown in Figure 6, query module specifically includes point query submodule 61, scope Query submodule 62, TopK query submodule 63.

The overall schematic diagram of the query module 6 is shown in Figure 5. The query request is first sent to the metadata server where the top-level index table is located, and the query request is located to a certain metadata server group through the top-level index generation module 5, and the query request is forwarded to the corresponding The group index generation module 4 locates the query request to one or some metadata servers by querying the group index table, forwards the query request to the corresponding local index generation module 3, finally determines the metadata set to be queried, and returns the result of the condition.

Specifically: the point query sub-module 61 processes the user's point query request, given the multi-dimensional attribute of a certain metadata, and the query result returns the specific information of the metadata; the range query sub-module 62 processes the user's range query request, and given the multi-dimensional The scope of the attribute, the query result returns all metadata information satisfying the scope in the whole system; the TopK query sub-module 63 processes the user’s TopK query request, given a group of multi-dimensional attributes, and specifies the K value, the query result returns the whole system with the given K pieces of data with the closest multidimensional attributes.

In order to verify the feasibility and effectiveness of the system of the present invention, the system of the present invention is configured in a real environment, and related query operations are performed to verify its effect.

The hardware and software system of the system test of the present invention are as shown in table 1:

Table 1

The configuration process of the system of the present invention is as follows: at first, the trace file of test is distributed to each node; Then, each node operates metadata collection generation module 1 and data server grouping module 2, and 5 nodes are divided into in this test Three groups, the number of nodes in the group is 1, 2, 3 respectively; run the local index generation module 3 in each group, select a node in each group to store the group index table generated by the group index generation module 4 , and the top-level index table generated by the top-level index generation module 5 is also saved on this node.

For the user's query request, it is processed by the query module 6, first query the top-level index table, determine the request for all metadata server groups, query the group index table in the group, determine the request for all metadata servers, and finally determine through the local index table The collection of metadata to query. Through this process, a request is finally limited to one or some metadata collections, thus effectively improving the time efficiency of the query. Table 2 is a comparison of the average query time between the system of the present invention and the relational database system.

Table 2

Those skilled in the art can easily understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (8)

1. based on a multi-dimensional metadata management method for associate feature, it is characterized in that, comprise the following steps:

(1) in metadata server cluster, the metadata on every platform meta data server is divided according to associate feature, with generator data acquisition and set statistics file; This step comprises following sub-step:

(1-1) multidimensional property representing associate feature between metadata on every platform meta data server is determined;

(1-2) multidimensional property of metadata is configured to the input vector of regular length, this input vector is as the input value of position sensitive hash function;

(1-3) use identical position sensitive hash function to carry out Hash calculation to input vector, the cryptographic hash obtained is as the unique identification of metadata corresponding to this input vector;

(1-4) metadata with identical cryptographic hash is divided in same collection of metadata, and using this cryptographic hash as the numbering of this collection of metadata;

(1-5) dividing condition of metadata in collection of metadata is added up, to generate set statistics file; This set statistics file comprises collection of metadata numbering, metadata number, each dimension attribute mean value, each dimension attribute scope, and wherein collection of metadata Serial Number Range is 1,2,3 ..., N, N represent the length of Hash table in position sensitive hash function;

(2) according to set statistics file, division operation is carried out to metadata cluster, to generate the grouping of multiple meta data server and packet configuration file;

(3) according to set statistics file, on every platform meta data server, local index table is set up respectively; Local index table is for managing the collection of metadata on every platform meta data server, and in concordance list, each have recorded the collection of metadata numbering in set statistics file, and the metadata set of this collection of metadata numbering correspondence is combined in the memory address in disk;

(4) according to packet configuration file and set statistics file, in each meta data server grouping, group index table is set up respectively;

(5) according to group index table, the top layer concordance list of metadata server cluster is set up;

(6) receive the inquiry request from user, and inquire about top layer concordance list, group index table and local index table successively according to inquiry request, and return Query Result; Wherein user's inquiry request comprises an inquiry, range query and TopK inquiry.

2. multi-dimensional metadata management method according to claim 1, it is characterized in that, step (2) is specially, every platform meta data server builds a bit vector, the length of this bit vector is identical with the Hash table length that position sensitive hash function in step (1-3) uses, thereafter, hierarchical clustering algorithm is utilized to carry out cluster operation between meta data server according to the bit vector of all meta data servers Hamming distances between any two, to obtain the grouping of meta data server, the grouping number formed when cluster reaches lower limit, or the distance between grouping reaches the upper limit, then stop cluster operation, thus obtain multiple meta data server group, and result is kept in packet configuration file.

3. multi-dimensional metadata management method according to claim 1, it is characterized in that, step (4) is specially, for each grouping in packet configuration file, build corresponding group index table respectively, each in group index table records the information of collection of metadata on all meta data servers in this grouping, comprises collection of metadata numbering, the IP address of collection of metadata place meta data server, metadata number, each dimension attribute mean value, each dimension attribute scope.

4. multi-dimensional metadata management method according to claim 1, is characterized in that, the some query manipulation in step (6) specifically comprises the following steps:

(6-1-1) acceptance point inquiry request, determines the multidimensional property of the metadata that this inquiry request is corresponding, and utilizes position sensitive hash function to calculate the cryptographic hash of multidimensional property, and this cryptographic hash is the numbering of the collection of metadata needing inquiry;

(6-1-2) list item that query metadata set number is corresponding in top layer concordance list, to obtain the IP address of this metadata place meta data server grouping;

(6-1-3) corresponding meta data server is determined according to the IP address of meta data server grouping, and the list item that query metadata set number is corresponding in the group index table of this meta data server, to obtain the IP address of this metadata place meta data server;

(6-1-4) according to the IP address of meta data server found, the list item that query metadata set number is corresponding in the group index table of this meta data server, is combined in memory address in disk to obtain this metadata place metadata set;

(6-1-5) be combined in the memory address in disk according to the metadata set found, the collection of metadata that inquiry is corresponding, and return Query Result.

5. multi-dimensional metadata management method according to claim 1, is characterized in that, the range query operation in step (6) specifically comprises the following steps:

(6-2-1) range of receiving inquiry request, determine multidimensional property scope to be checked, calculate the median of each range of attributes, and construct input vector by the median of each range of attributes, utilize position sensitive hash function to calculate the cryptographic hash of input vector, this cryptographic hash is the numbering of the collection of metadata needing inquiry;

(6-2-2) list item that query metadata set number is corresponding in top layer concordance list, compares the multidimensional property scope of preserving in the multidimensional property scope in inquiry request and list item, if two scopes are non-intersect, directly returns results as sky; If two scopes intersect, obtain the IP address of the meta data server grouping comprising metadata to be checked;

(6-2-3) corresponding meta data server is determined according to the IP address of meta data server grouping, and the list item that query metadata set number is corresponding in the group index table of this meta data server, to obtain the IP address of this metadata place meta data server;

(6-2-4) according to the IP address of meta data server found, the list item that query metadata set number is corresponding in the group index table of this meta data server, is combined in memory address in disk to obtain this metadata place metadata set;

(6-2-5) be combined in the memory address in disk according to the metadata set found, the collection of metadata that inquiry is corresponding, and return all metadata meeting multidimensional property scope in inquiry request.

6. multi-dimensional metadata management method according to claim 1, is characterized in that, the TopK query manipulation in step (6) specifically comprises the following steps:

(6-3-1) receive TopK inquiry request, determine multidimensional property and the K value of the metadata that this inquiry request is corresponding, and utilize position sensitive hash function to calculate the cryptographic hash of multidimensional property, this cryptographic hash is the numbering of the collection of metadata needing inquiry; Wherein K represents the quantity of the metadata the most close with the metadata multidimensional property in inquiry request;

(6-3-2) list item that query metadata set number is corresponding in top layer concordance list, if the metadata number recorded in list item is less than K value, then also include the list item of this list item the right and left in query context, until metadata number sum is more than or equal to K value in list item, finally obtain the IP address of multiple meta data server grouping;

(6-3-3) corresponding meta data server is determined according to the IP address of meta data server grouping, and the list item that query metadata set number is corresponding in the group index table of this meta data server, to obtain the IP address of this metadata place meta data server;

(6-3-4) according to the IP address of meta data server found, the list item that query metadata set number is corresponding in the group index table of this meta data server, is combined in memory address in disk to obtain this metadata place metadata set;

(6-3-5) be combined in the memory address in disk according to the metadata set found, the collection of metadata that inquiry is corresponding, and return the K bar metadata the most close with the multidimensional property of the metadata in inquiry request.

7., based on a multidimensional meta data management system for associate feature, comprise collection of metadata generation module, meta data server grouping module, local index generation module, group index generation module, top layer index generation module, enquiry module, it is characterized in that,

Collection of metadata generation module divides according to associate feature the metadata on every platform meta data server, with generator data acquisition and set statistics file; Collection of metadata generation module comprises multidimensional property determination module, input vector constructing module, hash function computing module, associate feature division module and output module;

The multidimensional property of associate feature between multidimensional property determination module determination representation element data;

The multidimensional property of metadata is configured to the input vector of regular length by input vector constructing module, and this input vector is as the input value of position sensitive hash function;

Hash function computing module uses identical position sensitive hash function to carry out Hash calculation to input vector, and the cryptographic hash obtained is as the unique identification of metadata corresponding to this input vector;

Associate feature divides module and the metadata with identical cryptographic hash is divided in same collection of metadata, and using this cryptographic hash as the numbering of this collection of metadata;

The dividing condition of metadata in output module statistics collection of metadata, to generate set statistics file;

Meta data server grouping module, according to set statistics file, carries out division operation to metadata cluster, to generate the grouping of multiple meta data server and packet configuration file;

Local index generation module, according to set statistics file, sets up local index table respectively on every platform meta data server;

Group index generation module, according to packet configuration file and set statistics file, sets up group index table respectively in each meta data server grouping;

Top layer index generation module, according to group index table, sets up the top layer concordance list of metadata server cluster;

Enquiry module receives the inquiry request from user, and inquires about top layer concordance list, group index table and local index table successively according to inquiry request, and returns Query Result.

8. multidimensional meta data management system according to claim 7, is characterized in that,

Enquiry module specifically to comprise some an inquiry submodule, range query submodule, TopK inquire about submodule;

The point inquiry request of some inquiry submodule process user, the multidimensional property of certain metadata given, Query Result returns the specifying information of metadata;

The range query request of range query submodule process user, the scope of given multidimensional property, Query Result returns in whole system all metadata informations meeting scope;

TopK inquires about the TopK inquiry request of submodule process user, given one group of multidimensional property, and refers to defining K value, and Query Result returns K bar data the most close with given multidimensional property in whole system.