CN106991137B - A method for indexing time series data based on Hbase hash summary forest - Google Patents

Abstract

The invention discloses a method for indexing time series data based on an Hbase hash summary forest, comprising the following steps: (1) establishing each time unit tree according to the time granularity; (2) obtaining the hash of each time unit tree code, and form the time unit tree with hash code into Hbase-based hash summary forest; (3) insert the collected time series data into the hash summary forest according to the hash code; (4) query and read according to the time range Get the stored time series data. The present invention improves the query speed of time-series data aggregation operation by combining the outline forest tree index scheme, and at the same time provides hash indexes for unit trees by generating hash codes, so as to solve the hotspot problem of Hbase distributed storage time-series data.

Description

A method for indexing time series data based on Hbase hash summary forest

technical field

The invention relates to the field of storage technology, in particular to a method for indexing time series data based on an Hbase hash summary forest.

Background technique

Time series data is continuous data indexed by time series. With the popularization of computer applications, time series data has also been widely used in various fields. For example: as the financial field is more and more closely integrated with the Internet, a large number of quantitative retracement operations in the financial field require more and more performance for the aggregation operation of time series data. For example: collect statistics on the market price, order price or trading volume of a certain commodity contract within a quarterly time range in futures, and perform aggregation operations such as summation or calculation of the maximum value. Such application scenarios frequently appear in financial quantification, and due to the huge amount of data, how to quickly and accurately calculate the aggregation operation results of financial time series data within t1~t2 becomes very important.

Take the summation operation of market prices within a certain time range in Au metal futures trading data as an example:

Select SUM(Last Price)From‘Au’WHERE time>t1 AND time<t2

In such an application scenario, it is necessary to quickly obtain aggregation operation results from massive time-series data.

Traditional relational databases mainly use materialized views or summary tables to speed up aggregation queries. The materialized view is to preprocess the query command involving table connection, and save the result in the view table, and directly fetch the preprocessed result when querying. The summary table calculates and saves the corresponding summary information while writing data, so that when a query occurs, the query is directly queried from the summary table and the result is returned. This type of method improves query efficiency, but the disadvantage is that it increases the expansion rate of the database. In NoSQL databases, some databases use MapReduce and aggregation pipelines to process these aggregation operations, which are representative of real-time computing. Although the expansion rate of the database is not increased, a large amount of disk and computing overhead is generated during the query process, which is low. It is time-consuming and cannot meet the needs of ad hoc queries. Some NoSQL databases integrate tree index structures to improve query efficiency and reduce the number of disk accesses.

Contents of the invention

In view of the above, the present invention proposes a method for indexing time-series data based on the Hbase hash summary forest, which speeds up the query time of time-series data by establishing a tree index, and avoids the time-series data in the distributed database through hash codes. The problem of uneven space allocation caused by sequential storage.

A method for indexing time series data based on Hbase hash summary forest, comprising the following steps:

(1) Establish each time unit tree according to the time granularity;

(2) Obtain the hash code of each time unit tree, and form the time unit tree with the hash code into an Hbase-based hash summary forest;

(3) Insert the collected time series data into the hash summary forest according to the hash code;

(4) Query and read the stored time series data according to the time range.

In step (1), the process of establishing a time unit tree is as follows: first, predetermine the time granularity of the time unit tree; then start recursion with the root node, and create a new node each time, and then recursively establish the left and right sides of this node Child nodes, stop recursion when the created nodes exceed the pre-calculated range, and complete the establishment process of the entire tree.

In step (1), each time unit tree is a line segment tree and contains a fixed time granularity. Time granularity is controlled by controlling the tree height of each tree. The line segment tree node stores the summary information of the node range, mainly including: LBound, RBound, LNode, RNode and Data; among them, LBound and RBound respectively represent the start time point and end time point of the node containing the time range; LNode and RNode respectively Indicates that the left and right child nodes of the node contain the midpoint of the time point; Data indicates the summary data value stored in the node, and the Data of each node of the time unit tree established at this time is empty.

The root node of each tree represents the aggregation result of time-series data within the time length of t carried by this tree, and the aggregation result of time-series data within the time length of t/2 is carried by the second-level nodes. By analogy, each layer of nodes carries half the time of the upper-layer nodes The length of the index data. By controlling the height of the tree to achieve a time unit tree with a fixed time granularity, it is convenient to use the same hash code to aggregate the nodes of each tree to achieve load balancing of hotspots.

A time unit tree represents the aggregation result of a unit time granularity range, and the leaf node of each tree represents the aggregate summary information of the finest granularity range. The granularity can be adjusted according to actual needs.

Time unit tree coverage time (TreeBound) calculation formula:

TreeBound＝(2^(TreeMaxLevel-1))*Leaf Bound

Among them, TreeMaxLevel is the maximum tree height, and LeafBound is the time range represented by the leaf node. For example: the time range of a time unit tree is set as one day, the tree height is set as 9 layers, and the leaf nodes represent 6-minute summary information. Therefore, a tree governs 1536 minutes of aggregated data, and a time unit tree covers the time range of one day.

In step (2), there are many ways to obtain the hash code of each time unit tree. As a preference, the corresponding hash code ( Hash), and write it into the tree hash table; the specific way to calculate the hash code is:

Hash＝md5(tree Info+tree low bound)

The tree info is the brief information of the data identification, for example: if this data represents the Au metal element futures data, it is marked as Au, and the tree low bound is the starting time point of the time unit tree.

Multiple time unit trees form a hash summary forest, and the time range of the data indexed by the entire hash summary forest is the sum of the time ranges represented by the time unit trees that make up it.

The Hbase-based hash summary forest consists of two Hbase tables, tree-hash and tree-node, where the tree-hash table is used to find the hash code of the time unit tree, and the tree-node table stores all the time unit tree trees node, and the tree-hash table and tree-node table are stored separately, that is, the hash code of each time unit tree is stored separately from the time series data carried by each time unit tree, and the time with the same hash code The time series data carried by the cell tree are stored centrally.

In step (3), the time-series data may be any time-series data such as financial time-series data and futures trading data.

In step (3), the specific process of inserting time series data into the hash summary forest is as follows:

(3-1) Find the tree hash value of the time unit tree where the time series data is located in the tree-hash table through the time to which the time series data belongs;

(3-2) Find the tree-node table corresponding to the tree hash value, and recursively insert the time series data into the tree-node table. The specific process is:

First, find the root node of the time unit tree according to the tree hash value and start recursion, and then compare the time point of the time series data with the time of the current query node. When the time point of the time series data is less than the time of the node, the The left child recursively inserts time series data. When the time point of the time series data is greater than the time of the node, the right child of the node recursively inserts time series data until it is inserted into the leaf node of the time unit tree.

In step (4), when data is queried, because the rowkey of the data in the same time unit tree has the same hash code, the hbase range query can quickly find out the entire time unit tree and store it in the hash summary forest In-memory, this greatly saves disk IO operations after recursing the tree.

The process of data query is:

(a) Determine whether the query time range (t1, t2) belongs to the same time unit tree range, if so, execute step (b), if not, execute step (c);

(b) The query operation is Query(t1,t2);

(c) The query operations are Query(t1,EndUnitTime(t1)), Query(midUnitTime) and Query(StartUnitTime(t2),t2);

Wherein, StartUnitTime(t1) is the starting time point of the time unit where the time point t1 is located;

EndUnitTime(t2) is the end time point of the time unit where the time point t2 is located;

midUnitTime is the time range of several unit trees between the time range of the time unit tree where t1 and t2 are located;

Query(t1, t2) is expressed as an execution query operation of querying the time range (t1, t2) in the same time unit tree;

Query(t1,EndUnitTime(t1)) indicates that the query operation is performed in the first unit tree in the query scope;

Query(midUnitTime) indicates that the query operation is performed in the second to penultimate unit trees in the query range;

Query(StartUnitTime(t2), t2) means to execute the query operation in the last unit tree in the query range.

The specific process of Query(t1,t2) is:

(a) Calculate the hash code of the time unit tree to which it belongs through the time of the search item and locate the time unit tree, and use the root node of the time unit tree as the current root node;

(b) Start recursive query from the current root node, and the query task starts from (t1, t2);

(c) When the current node is recursively queried, the starting time point LBound, the middle time point midTime and the ending time point RBound of the time range included in the node are parsed out;

(d) Judging whether t1 and t2 satisfy t1=LBound and t2=RBound, if so, record the node result and exit recursion, if not, execute step (e);

(e) Judging whether t1 and t2 satisfy t1≤midTime and t2≤midTime, if so, take the left child node of the node as the current root node, skip to step (b) to step (d), if not, perform step ( f);

(f) Determine whether t1 and t2 satisfy t1≥midTime and t2≥midTime, if so, take the right child node of the node as the current root node, and perform steps (b) to (d); if not, perform step (g) ;

(g) Determine whether the time t2 satisfies LBound<t2<midTime, if so, take the left child as the current node, midTime as t2, and execute steps (b) to (d); take the right child node as the current node, and midTime as t1, execute step (b) to step (d).

The present invention improves the query speed of time-series data aggregation operation by combining the outline forest tree index scheme, and at the same time provides hash indexes for unit trees by generating hash codes, so as to solve the hotspot problem of Hbase distributed storage time-series data.

Description of drawings

Fig. 1 is the flow chart of the method that time series data is indexed based on Hbase hash general forest of the present invention;

Fig. 2 is the structural representation of the established time unit tree;

FIG. 3 is a comparison chart of write data throughput in Embodiment 1 of the present invention;

FIG. 4 is a time-consuming comparison diagram of large-span time range aggregation query in Embodiment 2 of the present invention;

FIG. 5 is a comparison diagram of query time consumption between the hash summary deep forest method and the non-hash summary deep forest method in Embodiment 3 of the present invention.

Detailed ways

In order to describe the present invention more specifically, the technical solutions of the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 1, the specific steps of the method for indexing time series data based on the Hbase hash summary forest are:

Step 1, establish each time unit tree: first, predetermine the time range of the time unit tree; then start recursion with the root node, and create a new node each time, and then recursively establish the left and right child nodes of this node, when Stop the recursion when the created node exceeds the pre-calculated range, complete the establishment process of the whole tree, and write each node into the tree node table, and the established time unit tree is shown in Figure 2;

In this step, each time unit tree established is a line segment tree and contains a fixed time granularity. Time granularity is controlled by controlling the tree height of each tree. The line segment tree node stores the summary information of the node range, mainly including: LBound, RBound, LNode, RNode and Data; among them, LBound and RBound respectively represent the start time point and end time point of the node containing the time range; LNode and RNode respectively Indicates that the left and right child nodes of the node contain the midpoint of the time point; Data indicates the summary data value stored in the node, and the Data of each node of the time unit tree established at this time is empty.

The root node of each tree represents the aggregation result of time-series data within the time length of t carried by this tree, and the aggregation result of time-series data within the time length of t/2 is carried by the second-level nodes. By analogy, each layer of nodes carries half the time of the upper-layer nodes The length of the index data. By controlling the height of the tree to achieve a time unit tree with a fixed time granularity, it is convenient to use the same hash code to aggregate the nodes of each tree to achieve load balancing of hotspots.

A time unit tree represents the aggregation result of a unit time granularity range, and the leaf node of each tree represents the aggregate summary information of the finest granularity range. The granularity can be adjusted according to actual needs.

Time unit tree coverage time (TreeBound) calculation formula:

TreeBound＝(2^(TreeMaxLevel-1))*Leaf Bound

Among them, TreeMaxLevel is the maximum tree height of the tree, and LeafBound is the time range represented by the leaf node.

Step 2, obtain the hash code of each time unit tree, and form the time unit tree with the hash code into an Hbase-based hash summary forest;

In this step, the specific way to calculate the hash code is:

Hash＝md5(tree Info+tree low bound)

tree info is the brief information of the data identification, for example, if this data represents Au metal element futures data, it will be marked as Au; tree low bound is the starting time point of the time unit tree, and the obtained Hash value will be written into the tree hash table;

Step 3, insert the collected time series data into the hash summary forest, the insertion process is:

Step 3-1, find the tree hash value of the time unit tree where the time series data is located in the tree-hash table through the time to which the time series data belongs;

Step 3-2, find the tree-node table corresponding to the tree hash value, and recursively insert the time series data into the tree-node table, the specific process is:

First, find the root node of the time unit tree according to the tree hash value and start recursion, and then compare the time point of the time series data with the time of the current query node. When the time point of the time series data is less than the time of the node, the The left child recursively inserts time series data. When the time point of the time series data is greater than the time of the node, the right child of the node recursively inserts time series data until it is inserted into the leaf node of the time unit tree.

Step 4. Query and read the stored time series data according to the time range. The query process is:

Step 4-1, determine whether the query time range (t1, t2) belongs to the same time unit tree range, if so, execute step 4-2, if not, execute step 4-3;

Step 4-2, the query operation is Query(t1,t2);

In step 4-3, the query operations are Query(t1, EndUnitTime(t1)), Query(midUnitTime) and Query(StartUnitTime(t2), t2).

Example 1

Utilize the method of the present invention, Opentsdb open source time series database method and original Hbase method to carry out data writing to same time series data, and record the writing throughput of every kind of method, as shown in Figure 3, can obtain the method of the present invention from Figure 3 because There are indexes (hash codes) and tree building processes, so the data writing speed is slower than the original Hbase method directly storing the original data, but faster than the opentsdb open source time series database method.

Table 1 shows the division of regions in Hbase when data is written by the method of the present invention.

It can be seen from Table 1 that the hash summary forest is split into multiple regions according to the hash value after triggering Hbase's region split. The same time unit tree with the same hash value will be divided into the same region. The hash value is randomly generated, and the new tree is evenly loaded and hashed in different regions, avoiding the hot spot problem.

Example 2

Utilize the method of the present invention, the Opentsdb open source time series database method and the original Hbase method to carry out the aggregation query operation of the data in a large span time range, and Fig. 4 is a time-consuming comparison chart of the aggregation query in the long-term range of the three methods, and can be obtained from Fig. 4 in large In the aggregation query operation spanning the time range, the method of the present invention performs better. At the same time, it can be seen that the original Hbase method takes tens of seconds to query the aggregation results of 200,000 pieces of data, which cannot meet the needs of fast ad hoc query. The Opentsdb open source time series database method is greatly improved due to the cache and index mechanism. The method of the invention is faster than the Opentsdb query speed, and with the increase of the query range, the query time-consuming increase is slower than other solutions. Explain that using the rowkey design of hbase and using range query to find out the entire tree and put it in memory can optimize query performance and save disk overhead.

Example 3

Using the method of the present invention and the non-hashing method to aggregate and query data, Fig. 5 is a time-consuming comparison chart of the hash summary forest method and the non-hashing method. From Figure 5, it can be seen that the rowkey of the non-hashing scheme lacks a hash field, and the query search line segment tree performs a random query on hbase every time it recurses. The same line segment tree may be scattered in different regions, and the query time consumption of the method of the present invention is shorter than that of the non-hashing method, and the advantage becomes more obvious as the query range increases.

The above-mentioned specific embodiments have described the technical solutions and beneficial effects of the present invention in detail. It should be understood that the above-mentioned are only the most preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, supplements and equivalent replacements made within the scope shall be included in the protection scope of the present invention.

Claims (4)

1. a kind of method being indexed based on Hbase hash summary forest to time series data, comprising the following steps:

(1) time range that high, leaf node includes according to tree establishes every time quantum tree comprising set time granularity；

(2) hash code of every time quantum tree is sought, and the time quantum tree with hash code is formed into dissipating based on Hbase Column summary forest；

(3) time series data of acquisition is inserted into hash summary forest according to hash code；

(4) time series data for reading storage is inquired according to time range；

Every time quantum tree is a Kd-Trees, and line segment tree node stores the summary info of the range of nodes, comprising: LBound, RBound, LNode, LNode and Data；Wherein, it includes time model that LBound, RBound, which respectively indicate the node, The start time point enclosed and termination time point；LNode, LNode respectively indicate the left child of the node and when right child nodes include Between range midpoint；Data indicates the summary data value of node storage, each node for the time quantum tree established at this time Data is empty；

The hash code Hash's of every time quantum tree seeks formula are as follows:

Hash=md5 (tree Info+tree low bound)

Tree info is Data Identification brief information；Tree low bound is the start time point of time quantum tree；Md5 is A kind of transcoding mode；

The hash summary forest based on Hbase is made of two Hbase tables of tree-hash and tree-node, wherein For tree-hash table for storing the corresponding hash code of all time quantum trees, each tree-node table has corresponding time quantum All leaf nodes of tree, and tree-hash table is individually to store, and will possess same Hash code with tree-node table The time series data that time quantum tree is loaded with is centrally stored.

2. the method being indexed according to claim 1 based on Hbase hash summary forest to time series data, feature are existed In: time series data is inserted into the detailed process of hash summary forest are as follows:

Time where (3-1) finds this timing data by the belonging time of time series data in tree-hash table is single The hash code of member tree；

(3-2) finds tree-node table corresponding to the hash code, and time series data recurrence is inserted into this tree-node table, Detailed process are as follows:

Start recurrence according to the root node that hash code finds locating time quantum tree, then the time point of progress time series data with work as The time range of preceding query node compares, and is less than the middle time point of the time range of the node when the time point of time series data When, into the Data of the left child nodes of the node, recurrence is inserted into time series data, is greater than the node when the time point of time series data Time range middle time point when, then in the Data of the right child nodes of the node recurrence be inserted into time series data；Until inserting Enter until the leaf node of time cell tree.

3. the method being indexed according to claim 1 based on Hbase hash summary forest to time series data, feature are existed In: carry out the process of data query are as follows:

(a) judge whether query time range (t1, t2) belongs to cell tree range at the same time, if so, step (b) is executed, If it is not, executing step (c)；

(b) inquiry operation Query (t1, t2) is executed

(c) inquiry operation Query (t1, EndUnitTime (t1)), Query (midUnitTime) and Query are executed (StartUnitTime(t2),t2)；

Wherein, StartUnitTime (t2) is the initial time for the time range that time quantum tree locating for time point t2 includes Point；

EndUnitTime (t1) is the end time point for the time range that time quantum tree locating for time point t1 includes；

MidUnitTime is the time range between the time range of time quantum tree locating for t1 and t2；

Query (t1, t2) is expressed as executing inquiry operation in the same time quantum tree belonging to query context t1~t2；

Query (t1, EndUnitTime (t1)) indicates first unit in query context t1~EndUnitTime (t1) Inquiry operation is executed in tree；

Query (midUnitTime) indicates second to the second from the bottom cell tree in query context midUnitTime Middle execution inquiry operation；

Query (StartUnitTime (t2), t2) indicates last in query context StartUnitTime (t2)~t2 Inquiry operation is executed in cell tree.

4. the method being indexed according to claim 3 based on Hbase hash summary forest to time series data, feature are existed In: the detailed process of Query (t1, t2) are as follows:

(a) go out the hash code of its affiliated time quantum tree by searching for the time reckoning of item and navigate to this time cell tree, and Using the root node of this time cell tree as current root node；

(b) recursive query since current root node, query task start from (t1, t2)；

(c) when recursive query is to present node, start time point LBound, the centre of the time range that the node includes are parsed Time point midTime and termination time point RBound；

(d) judge whether t1 and t2 meet t1=LBound and t2=RBound, passed if so, recording the node result and exiting Return, if it is not, executing step (e)；

(e) judge whether t1 and t2 meet t1≤midTime and t2≤midTime, if so, the left child nodes of the node are made It for current root node, jumps and executes step (b)~step (d), if it is not, executing step (f)；

(f) judge whether t1 and t2 meet t1 >=midTime and t2 >=midTime, if so, the right child nodes of the node are made For current root node, step (b)~step (d) is executed；If it is not, executing step (g)；

(g) judge whether time t2 meets LBound < t2 < midTime, if so, using left child as present node, it will MidTime executes step (b)~step (d) as t2；Using right child nodes as present node, using midTime as t1, Execute step (b)~step (d).