CN106250457A - The inquiry processing method of big data platform Materialized View and system - Google Patents

Abstract

The invention discloses inquiry processing method and the system of big data platform Materialized View, the selection step of Materialized View: generate MVPP structure chart for given query set, the set of all nonleaf nodes is obtained according to this structure chart, calculate the value of each nonleaf node in this set, utilize the Materialized View greed selection algorithm towards maximum revenue to obtain Materialized View set；The placement step of Materialized View: Materialized View associated weights matrix is set up in the Materialized View set for the selection step acquisition of Materialized View, calculate the value of each element in matrix, by all calculating nodes by the descending by size of materialization memory space, the node that materialization memory space is maximum is taken, it is thus achieved that be placed into the Materialized View of this node in all calculating nodes.The Materialized View laying method of the present invention can reduce internodal network data transmission, and shortening processes the time.

Description

The inquiry processing method of big data platform Materialized View and system

Technical field

The present invention relates to big data query process field, be specifically related to the inquiry processing method of big data platform Materialized View And system.

Background technology

Deepening continuously along with intelligent grid construction and advance, the data volume that operation of power networks and monitoring of equipment produce exponentially increases The data that long, power grid enterprises can store, process have reached unprecedented magnitude, and to exceed the speed of Moore's Law Swift and violent increase.The value that data are most crucial greatly is to store for mass data and analyze exactly.In commercial environments, number According to processing service provider, the process of big data is packaged into service, is sold to user.Along with the development of big data commercial applications, Its data process and occur in that following characteristics:

1) big data platform number of users is huge.Big data service provider is wished for more customer service, to obtain More commercial profit.

2) user presents variation feature to the performance requirement of big Data Analysis Services.Demand is analyzed in some user's real time implementations More, and the time of return that some users are to inquiring about is not to take notice of especially.For the data analysis requirements that some are real-time, user The time of the performance processed and return is had required.Common practice is, user signs one with big data service provider Agreement about time of return.The most early, the income of service provider is the highest for time of return；Otherwise, time of return is the slowest, and income is more Low, even need to support certain chastening amount of money.

3) big data platform needs to process a large amount of analytical inquiry that a large number of users is initiated, and therefore it needs to rely on cloud computing Platform completes the process of various data and affairs.

Big data platform can service numerous user to be provided Data Analysis Services service and brings profit for service provider. But, in the analysis inquiry of big data platform, there is a large amount of attended operation, this operation is relatively time-consuming.That a part is commonly used, Crucial connection intermediate object program stores in external memory, it is possible to use attended operation is avoided in inquiry, thus reaches to shorten return The purpose of time.The data of our this intermediate object program are called Materialized View.It is bigger that Materialized View takies memory space, although big Data platform has bigger memory space, but can not provide unlimited Materialized View memory space.Therefore we need respectively View selects, and conventional, crucial view are carried out materialization.Above-mentioned big data platform is some servers composition Cloud computing environment, substantially server cluster.

In the structure cloud computing platform of Share-Nothing, each calculating node possesses storage function, each node It is required for retaining certain memory space storage Materialized View.The Materialized View chosen needs to be placed in each node.Different Laying method, can to analyze inquiry have different effect of optimizations.Patent the most of the present invention also provides for putting of a kind of Materialized View Put method.

Summary of the invention

For solving the deficiency that prior art exists, the invention discloses the inquiry processing method of big data platform Materialized View And system, the inquiry processing method of big data platform Materialized View includes embodied spillover method, Materialized View laying method. The system of selection of Materialized View is the basis of Materialized View laying method, and the Materialized View algorithm that is chosen as of Materialized View provides defeated Enter target, i.e. input needs the Materialized View set placed, it is possible to reduce internodal network data transmission, when shortening processes Between.

For achieving the above object, the concrete scheme of the present invention is as follows:

The inquiry processing method of big data platform Materialized View, including:

The selection step of Materialized View: generate MVPP structure chart for given query set, obtain according to this structure chart The set of all nonleaf nodes, calculates the value of each nonleaf node in this set, utilizes the materialization towards maximum revenue to regard Figure greed selection algorithm obtains Materialized View set；

The placement step of Materialized View: the Materialized View set obtained for the selection step of Materialized View is set up materialization and regarded Figure associated weights matrix, calculates the value of each element in matrix, by all calculating nodes by materialization memory space descending by size Arrangement, takes the node that materialization memory space is maximum, it is thus achieved that be placed into the Materialized View of this node in all calculating nodes.

Further, for given query set, utilize MVPP algorithm to utilize and generate MVPP structure chart, by oriented nothing The form of ring figure states the overall query process tactic for query set.

Further, in MVPP structure chart, with leaf node represent be in data base the fact table, root node then table Show is that all nonleaf nodes can serve as the selection object of Materialized View based on the fact that the inquiry of table.

Further, if E is the set of all nonleaf nodes in MVPP structure chart, in MVPP structure chart, a n omicronn-leaf Node e_jRepresent, each nonleaf node e_jAll corresponding candidate's Materialized View, uses m_jRepresent e_jThe candidate that nonleaf node represents Materialized View.

Further, nonleaf node e_jValue υ_jRepresenting, its computational methods are:

$v_j=\underset{q_i\∈Q_j}{\Σ}\[a_i-b_i\×t_{i,j}-(a_i-b_i\×t_i)\]=\underset{q_i\∈Q_j}{\Σ}\[b_i\×(t_i-t_{i,j})\]$

t_{I, j}For working as e_jCorresponding candidate Materialized View m_jQ during materialization_iTime of return, t_{I, j}Obtained by test, Q_j For m can be relied on_jComplete the query set of query processing, a_iFor initial return, i.e. one inquiry q_iThe receipts that can return immediately Benefit, b_iFor punishment slope, it represents the speed that passage income declines in time, t_iIt is an inquiry q_iTime of return.

Further, the Materialized View greed selection algorithm towards maximum revenue is selection based on valueization MVPP figure Algorithm, its input is the MVPP figure of the value calculating each nonleaf node, is output as a Materialized View set.

Further, towards the concretely comprising the following steps of Materialized View greed selection algorithm of maximum revenue:

1) set F as a Materialized View set, and be initialized as sky；

2) υ during calculating obtains E_j/s_jMaximum nonleaf node e_j, wherein s_jFor e_jDepositing shared by corresponding candidate's Materialized View Storage space size, nonleaf node e_jValue υ_jRepresent；

3) by e_jCorresponding candidate's Materialized View adds F, i.e. to m_jCarrying out materialization, Formal Representation is: F ← F ∪ m_j, E ← E-e_j；

4) value of other nonleaf nodes in E is updated；

5) if the memory space sum that the Materialized View of F takies is less than the total storage space S of system Materialized View, then repeat to hold Row the 2nd) step；Otherwise terminating algorithm, F is the set of required Materialized View.

Further, described step 4) in concrete renewal operating procedure:

To responsible m_jPerform query set Q_j, for Q_jIn each inquiry q_iPerform following operation: q_iTime of return t_i It is updated to based on m_jPerform the time of return of query processing；

Due to materialization m_jAfter, e can be affected_jDescendants's nonleaf node and the value of ancestors' nonleaf node, according to nonleaf node e_j's Value computing formula recalculates e_jThe value of descendants's nonleaf node and ancestors' nonleaf node.

Further, in Materialized View associated weights matrix H, H has | F | row to arrange with | F |, if h_{I, j}For an element of H, So h_{I, j}Then represent m_iWith m_jThe associated weights of two Materialized Views, the quantity of | F | Materialized View to be placed.

Further, calculating the value of each element in H, its computational methods are:In above formula, b_kIt is Refer to q_kPunishment slope in revenue function；Q_{I, j}Refer to need to access in Q m when query processing simultaneously_iWith m_jTwo materializations regard The set of the inquiry of figure.

Further, representing that cloud is fallen into a trap operator node set with N, | N | is the quantity calculating node, designs operator node n_kThing Changing view storage size is s_k, take, at N, the n that materialization memory space is maximum_k, the thing of this node it is placed into by calculating acquisition Changing view, computational methods are:

A) in H, row and maximum Materialized View m are selected from | F | row_iAs initial cluster center；

B) by m_iIt is placed into n_k；

C) take out successively and m at F_iThe m that relation weight is maximum_j, by m_jIt is placed into n_k, until being placed into n_kMaterialized View account for Memory space more than s_k。

Further, after obtaining the Materialized View step being placed into this node, perform N ← N-n_kStep, then weighs The multiple step performing to be obtained the Materialized View being placed into this node by calculating, until

The query processing system of big data platform Materialized View, including:

The selection module of Materialized View: generate MVPP structure chart for given query set, obtain according to this structure chart The set of all nonleaf nodes, calculates the value of each nonleaf node in this set, utilizes the materialization towards maximum revenue to regard Figure greed selection algorithm obtains Materialized View set；

The placement module of Materialized View: the Materialized View set obtained for the selection step of Materialized View is set up materialization and regarded Figure associated weights matrix, calculates the value of each element in matrix, by all calculating nodes by materialization memory space descending by size Arrangement, takes the node that materialization memory space is maximum, it is thus achieved that be placed into the Materialized View of this node in all calculating nodes.

Beneficial effects of the present invention:

The present invention provides a kind of big data embodied spillover method and Materialized View laying method in cloud, maximizes big The income of data platform service provider.

The present invention has quantified the impact on income of each Materialized View, and the selection offer for Materialized View is supported accurately.

The Materialized View laying method of the present invention can reduce internodal network data transmission, and shortening processes the time.

The selection of the Materialized View of the present invention and placement can adapt to large-scale data processing environment, and are prone at cloud meter Calculate in environment and extend.

The present invention is adapted to the scene of each node materialization storage size isomery in cloud.

Accompanying drawing explanation

Fig. 1 MVPP structure chart；

The selection of the big data platform Materialized View of Fig. 2 present invention and placement flow chart.

Detailed description of the invention:

The present invention is described in detail below in conjunction with the accompanying drawings:

As in figure 2 it is shown, the inquiry processing method of the big data platform Materialized View of patent of the present invention is embodied as main point It is two parts: embodied spillover method and Materialized View laying method.

One, embodied spillover method:

1) because the big data query in each cycle is varied from, therefore the cycle carry out embodied spillover.

2) in cloud computing node, one is arbitrarily selected to calculate node as host node, looking into of its collection upper cycle Ask set.

3) query set is generated MVPP structure chart.All query processing plan (Multi-View Processing Plan, is abbreviated as MVPP), state the overall query process tactic for query set by the form of directed acyclic graph.In Fig. 1 With leaf node represent be in data base the fact table, what root node then represented is based on the fact that the inquiry of table.All n omicronn-leaf Node can serve as the selection object of Materialized View.

In FIG, as a example by Q1, Q2, Q3, Q4, Q5, present the MVPP structure chart that they generate.As inquire about Q3 be by Table I tem, Sale and Part connect and obtain, i.e. (Item ∞ Sale) ∞ Part.When inquiring about quantity and being more, such possible Inquiry plan is the most, and nonleaf node can expand quickly.Therefore partial view can only be carried out materialization.By query set symphysis Become MVPP structure chart referring specifically to document: J.Yang, K.Karlapalem, and Q.Li, " Algorithms for materialized view design in data warehousing environment,"in VLDB,1997, pp.136-145.

4) formula (3) is utilized to calculate the value of each nonleaf node.

If E is the set of all nonleaf nodes in MVPP structure chart.In MVPP, a nonleaf node e_jRepresent.Often Individual nonleaf node e_jAll corresponding candidate's Materialized View, uses m_jRepresent e_jCandidate's Materialized View that nonleaf node represents.

The selection of the Materialized View turning to target with Income Maximum is closely related with the revenue function of inquiry with laying method. First the revenue function analyzing inquiry is given.One inquiry q_iRepresenting, i is the numbering of this inquiry.If the income of an inquiry Function is R, then according to q_iParameter calculate R method be

R(t_i)=a_i-b_i×t_i (1)

In equation 1, a_iFor initial return, i.e. q_iThe income that can return immediately.b_iFor punishment slope, it represents at any time Between pass income decline speed.t_iFor q_iTime of return.

Nonleaf node e_jValue υ_jRepresenting, its computational methods are:

$v_j=\underset{q_i\∈Q_j}{\Σ}R\left(t_{i,j}\right)-R\left(t_i\right)---\left(2\right)$

It is derived as further according to formula (1):

$v_j=\underset{q_i\∈Q_j}{\Σ}\[a_i-b_i\×t_{i,j}-(a_i-b_i\×t_i)\]=\underset{q_i\∈Q_j}{\Σ}\[b_i\×(t_i-t_{i,j})\]---\left(3\right)$

t_{I, j}For working as e_jCorresponding candidate Materialized View m_jQ during materialization_iTime of return, t_{I, j}Obtained by test.Q_j For m can be relied on_jComplete the query set of query processing.

5) the Materialized View greed selection algorithm towards maximum revenue is utilized to obtain Materialized View set.Towards income The Materialized View greed selection algorithm of bigization is a kind of based on valueization MVPP figure the selection algorithm that patent of the present invention proposes, its Input is the MVPP figure of the value calculating each nonleaf node, is output as a Materialized View set, and its key step is as follows:

(1) set F as a Materialized View set, and be initialized as sky.

(2) υ during calculating obtains E_j/s_jMaximum nonleaf node e_j, wherein s_jFor e_jShared by corresponding candidate's Materialized View Storage size.

(3) by e_jCorresponding candidate's Materialized View adds F, i.e. to m_jCarry out materialization.Namely: F ← F ∪ m_j, E ← E-e_j。

(4) value of other nonleaf nodes in E is updated.Due to by m_jAfter materialization, the time of return of some inquiries can occur Change, thus cause the value of some nonleaf node to change, it is therefore desirable to it is worth and updates operation.

Hereinafter update the concrete steps of operation:

A) to responsible m_jPerform query set Q_j, for Q_jIn each inquiry q_iPerform following operation.q_iTime of return t_iIt is updated to based on m_jPerform the time of return of query processing.

B) due to materialization m_jAfter, the e that can affect_jDescendants's nonleaf node and the value of ancestors' nonleaf node.According to formula (3) Recalculate e_jThe value of descendants's nonleaf node and ancestors' nonleaf node.

(5) if the memory space sum that the Materialized View of F takies is less than the total storage space S of system Materialized View, then repeat Perform the 2nd) step；Otherwise terminating algorithm, F is the set of required Materialized View.

Two, Materialized View laying method

When carrying out query processing, two or above Materialized View may be used.When these inquiry view distributions When multiple node, larger data transmission cost can be produced, thus reduce query processing speed.Patent the most of the present invention proposes one Plant Materialized View Placement based on associated weights.Associated weights refers to what two Materialized Views were accessed by an inquiry simultaneously Frequency values.

Representing that cloud is fallen into a trap operator node set with N, | N | is the quantity calculating node.Design operator node n_kMaterialized View deposit Storage space size is s_k.The key step of Materialized View Placement based on associated weights is as follows:

1) master computing node is responsible for distributing Materialized View, and other calculate node and are responsible for receiving and storage Materialized View.

2) Materialized View associated weights matrix H is set up.H has | F | row to arrange with | F |, if h_{I, j}An element for H, then h_{I, j}Then represent m_iWith m_jThe associated weights of two Materialized Views.The quantity of | F | Materialized View to be placed.

3) value of each element in H is calculated.Such as h_{I, j}, its computational methods are:In above formula, b_kRefer to q_kPunishment slope in revenue function (formula (1))；Q_{I, j}Referring to some inquiries in Q, these inquiries, when processing, need Access m simultaneously_iWith m_jTwo Materialized Views.

4) by all calculating nodes by the descending by size of materialization memory space.

5) take, at N, the n that materialization memory space is maximum_k, the Materialized View of this node it is placed into by calculating acquisition.Calculating side Method is:

I., in H, row and maximum Materialized View m are selected from | F | row_iAs initial cluster center.

Ii. by m_iIt is placed into n_k。

Iii. take out successively and m at F_iThe m that relation weight is maximum_j.By m_jIt is placed into n_k, until being placed into n_kMaterialized View The memory space taken is more than s_k。

6)N←N-n_k, will node n_k, delete from both candidate nodes set.

7) the 5th is repeated) step, until

The invention also discloses the query processing system of big data platform Materialized View, including:

The selection module of Materialized View: generate MVPP structure chart for given query set, obtain according to this structure chart The set of all nonleaf nodes, calculates the value of each nonleaf node in this set, utilizes the materialization towards maximum revenue to regard Figure greed selection algorithm obtains Materialized View set；

The placement module of Materialized View: the Materialized View set obtained for the selection step of Materialized View is set up materialization and regarded Figure associated weights matrix, calculates the value of each element in matrix, by all calculating nodes by materialization memory space descending by size Arrangement, takes the node that materialization memory space is maximum, it is thus achieved that be placed into the Materialized View of this node in all calculating nodes.

The function that in the said system of the present invention, the selection module of Materialized View is realized relies on the choosing of above-mentioned Materialized View The concrete step of selection method and algorithm realize, and the realization of the function of the placement module of Materialized View relies on above-mentioned Materialized View The concrete step of laying method and algorithm realize.

Although the detailed description of the invention of the present invention is described by the above-mentioned accompanying drawing that combines, but not the present invention is protected model The restriction enclosed, one of ordinary skill in the art should be understood that on the basis of technical scheme, and those skilled in the art are not Need to pay various amendments or deformation that creative work can make still within protection scope of the present invention.

Claims (10)

1. the inquiry processing method of big data platform Materialized View, is characterized in that, including:

The selection step of Materialized View: generate MVPP structure chart for given query set, owned according to this structure chart The set of nonleaf node, calculates the value of each nonleaf node in this set, utilizes the Materialized View towards maximum revenue to covet Heart selection algorithm obtains Materialized View set；

The placement step of Materialized View: Materialized View pass is set up in the Materialized View set for the selection step acquisition of Materialized View Connection weight matrix, calculates the value of each element in matrix, by all calculating nodes by the descending by size of materialization memory space, The node that materialization memory space is maximum is taken, it is thus achieved that be placed into the Materialized View of this node in all calculating nodes.

2. the inquiry processing method of big data platform Materialized View as claimed in claim 1, is characterized in that, look into for given Ask set, utilize MVPP algorithm to utilize and generate MVPP structure chart, stated for query set by the form of directed acyclic graph Overall query process tactic.

3. the inquiry processing method of big data platform Materialized View as claimed in claim 2, is characterized in that, at MVPP structure chart In, with leaf node represent be in data base the fact table, what root node then represented is based on the fact that the inquiry of table, all non- Leaf node can serve as the selection object of Materialized View.

4. the inquiry processing method of big data platform Materialized View as claimed in claim 3, is characterized in that, if E is MVPP knot The set of all nonleaf nodes in composition, in MVPP structure chart, a nonleaf node e_jRepresent, each nonleaf node e_jAll Corresponding candidate's Materialized View, uses m_jRepresent e_jCandidate's Materialized View that nonleaf node represents.

5. the inquiry processing method of big data platform Materialized View as claimed in claim 1, is characterized in that, towards Income Maximum The Materialized View greed selection algorithm changed is selection algorithm based on valueization MVPP figure, and its input is to calculate each non-leaf segment The MVPP figure of the value of point, is output as a Materialized View set.

6. the inquiry processing method of big data platform Materialized View as claimed in claim 1, is characterized in that, towards Income Maximum Concretely comprising the following steps of the Materialized View greed selection algorithm changed:

1) set F as a Materialized View set, and be initialized as sky；

2) v during calculating obtains E_j/s_jMaximum nonleaf node e_j, wherein s_jFor e_jCorresponding storage shared by candidate's Materialized View is empty Between size, nonleaf node e_jValue v_jRepresent；

3) by e_jCorresponding candidate's Materialized View adds F, i.e. to m_jCarrying out materialization, Formal Representation is: F ← F ∪ m_j, E ← E-e_j；

4) value of other nonleaf nodes in E is updated；

5) if the memory space sum that takies of the Materialized View of F is less than the total storage space S of system Materialized View, then the is repeated 2) step；Otherwise terminating algorithm, F is the set of required Materialized View.

7. the inquiry processing method of big data platform Materialized View as claimed in claim 6, is characterized in that, described step 4) in Concrete renewal operating procedure:

To responsible m_jPerform query set Q_j, for Q_jIn each inquiry q_iPerform following operation: q_iTime of return t_iUpdate For based on m_jPerform the time of return of query processing；

Due to materialization m_jAfter, e can be affected_jDescendants's nonleaf node and the value of ancestors' nonleaf node, according to nonleaf node e_jValue E is recalculated by computing formula_jThe value of descendants's nonleaf node and ancestors' nonleaf node.

8. the inquiry processing method of big data platform Materialized View as claimed in claim 1, is characterized in that, Materialized View associates In weight matrix H, H has | F | row to arrange with | F |, if h_i,jAn element for H, then h_i,jThen represent m_iWith m_jTwo Materialized Views Associated weights, the quantity of | F | Materialized View to be placed.

9. the inquiry processing method of big data platform Materialized View as claimed in claim 1, is characterized in that, represent in cloud with N Calculating node set, | N |, for calculating the quantity of node, designs operator node n_kMaterialized View storage size be s_k, take at N The n that materialization memory space is maximum_k, the Materialized View of this node it is placed into by calculating acquisition, computational methods are:

A) in H, row and maximum Materialized View m are selected from | F | row_iAs initial cluster center；

B) by m_iIt is placed into n_k；

C) take out successively and m at F_iThe m that relation weight is maximum_j, by m_jIt is placed into n_k, until being placed into n_kMaterialized View take Memory space is more than s_k。

10. the query processing system of big data platform Materialized View, is characterized in that, including:

The selection module of Materialized View: generate MVPP structure chart for given query set, owned according to this structure chart The set of nonleaf node, calculates the value of each nonleaf node in this set, utilizes the Materialized View towards maximum revenue to covet Heart selection algorithm obtains Materialized View set；

The placement module of Materialized View: Materialized View pass is set up in the Materialized View set for the selection step acquisition of Materialized View Connection weight matrix, calculates the value of each element in matrix, by all calculating nodes by the descending by size of materialization memory space, The node that materialization memory space is maximum is taken, it is thus achieved that be placed into the Materialized View of this node in all calculating nodes.