CN104794239A - Cloud platform data processing method - Google Patents

Abstract

The invention provides a cloud platform data processing method. The method comprises the steps that a video data file in a cloud storage system is stored in a data partitioning and multi-version mode and planned for parallel affair processing, a plurality of data nodes in a cloud platform are clustered with a mapping relation between data blocks and versions as features, and affairs not executed are dynamically dispatched according to the feedback information of the executed affairs. According to the method, the resource utilization rate of the cloud platform system and the load balance effect are improved.

Description

A kind of cloud platform data disposal route

Technical field

The present invention relates to cloud computing, particularly a kind of cloud platform data disposal route.

Background technology

Cloud storage has the feature of high reliability, high scalability, cheap low expense.Every platform storing machine is all an independently memory node, and adding and the normal operation being separated not influential system of node, the data volume of file is large; In process massive video data, there is advantage.But, because video data deblocking quantity is a lot, and each deblocking and version thereof are stored in the different nodes in cloud platform, therefore in great deal of nodes, need the optimal scheduling carrying out affairs, and scheduling at present in cloud platform and resource allocation mechanism are one local scheduling mechanism statically, when carrying out Resource Distribution and Schedule, only consider the current resource status of range data nearest node, and do not carry out the optimal allocation of resource from the angle of the system overall situation.

Summary of the invention

For solving the problem existing for above-mentioned prior art, the present invention proposes a kind of cloud platform data disposal route, comprising:

Video data file in cloud storage system is stored with the form of deblocking multi version and plans parallel processing transaction, with the mapping relations between data block and version for feature, cluster is performed to back end multiple in cloud platform, carrys out the unenforced affairs of dynamic dispatching according to the feedback information of executed affairs.

Preferably, described by the video data file in cloud storage system with the form of deblocking multi version store, comprise further:

The user's request of Video Applications is submitted to the client of cloud storage system with the form of mission description file, when affairs are submitted, client is divided into multiple subtransaction according to deblocking information affairs and carrys out executed in parallel, each subtransaction associates a data block, perform so that different subtransaction is distributed on different nodes, described deblocking information is described by tree Tr, Tr=(C, R), wherein C is a data element set, and each element represents a data block; R is whether the content that an each element of binary relation set represents two data blocks is identical, the element c in described data element set C _ican describe with two tuple vectors: c _i=<block _i, a _i>, wherein block _irepresent the numbering of data block i, a _irepresent the numbering of data block i place node, the element r in described binary relation set R _{i, j}represent that data block j is the backup version of data block i.

Preferably, each described back end has a dispatching process, the dispatching process of different node communicates with the dispatching process of client according to the current state of the virtual machine of place back end, transmit transaction scheduling and control information, after affairs are divided into subtransaction according to the video data blocking information tree Tr inquired by the dispatching process of client, according to binary relation set R, back end is performed cluster, clustering rule is:

For arbitrary node a _i, a _i∈ D _mif, a _j∈ D _mand a _i≠ a _j, then there is r _{i, j}∈ R

Wherein D _mfor storing the cluster set of all back end compositions of same data block m, the dispatching process of client is at D _minterior selection optimal node is executed the task;

If the dispatching process of back end receives affairs, then these affairs insert the wait transaction queues of this node according to its priority, or the affairs seizing other low priority perform immediately, after affairs are finished, the dispatching process of back end just notifies to make the dispatching process of client carry out the scheduling of subsequent transactions by the dispatching process of client.

The present invention compared to existing technology, has the following advantages:

The method that the present invention proposes improves the resource utilization of cloud plateform system and the effect of load balancing.

Accompanying drawing explanation

Fig. 1 is the process flow diagram of the cloud platform data disposal route according to the embodiment of the present invention.

Embodiment

Detailed description to one or more embodiment of the present invention is hereafter provided together with the accompanying drawing of the diagram principle of the invention.Describe the present invention in conjunction with such embodiment, but the invention is not restricted to any embodiment.Scope of the present invention is only defined by the claims, and the present invention contain many substitute, amendment and equivalent.Set forth many details in the following description to provide thorough understanding of the present invention.These details are provided for exemplary purposes, and also can realize the present invention according to claims without some in these details or all details.

The present invention, under cloud computing environment, carries out modeling to the affairs of processing video data and transaction scheduling and resource allocation methods, proposes a kind of transaction scheduling method towards video data under this model.Parallel processing transaction is planned based on the piecemeal multi version file layout of the method video data file in cloud storage system, with the mapping relations of data block and version for feature performs cluster to back end in cloud platform, based on the feedback information of the complete affairs of executed, carry out dynamic dispatching do not perform affairs, improve the effect of resource utilization ratio and load balancing with this.Fig. 1 is the cloud platform data process flow figure according to the embodiment of the present invention.

User's request in Video Applications is submitted to the client of cloud storage system with the form of mission description file.When affairs are submitted, client is divided into multiple subtransaction according to deblocking information these affairs and carrys out executed in parallel.Each subtransaction associates a data block.In same like this affairs, different subtransaction just can be distributed on different nodes and perform, and can avoid the bottleneck caused due to resource contention between subtransaction, the feature that in adaptive video data handling procedure, resource consumption is high better.Deblocking information is described by tree Tr:

Tr=(C, R), wherein C is a data element set, and element wherein represents a data block; R is a binary relation set, and whether the content that element wherein represents two data blocks is identical.Element c in data element set C _ican describe with two tuple vectors: c _i=< _blocki, a _i>

Wherein _blockirepresent the numbering of data block i, a _irepresent the numbering of data block i place node.Element r in binary relation set R _{i, j}represent that data block j is the copy version of data block i.Be submitted to the client of cloud storage system in random time affairs, the transaction flow being submitted to client can be described as J={J ₀, J ₁, J ₂, J _i..., J _n-1, J _n}

Wherein j _ican describe with hexa-atomic group of vector: j _i=<transid _i, dec _id _i, file _id _i, level _i, rcv_t _i, end_t _i>, wherein transid is affairs numberings, dec _id is mission description reference number of a document, file _id is that affairs need video data file to be processed numbering, and level is the priority of affairs, and rcv_t is the submission time of affairs, and end_t has been the closing time of these affairs.The affairs j submitted to _imultiple subtransaction is divided into by client.Subtransaction after division can describe with a digraph GR:

GR＝<V，E>

Each element in vertex set V represents a subtransaction.For subtransaction v _idescribe with nine tuple vectors: v _i=<taskid _i, transid _i, type _i, wl _i, cpu _i, mem _i, disk _i, band _i, block _i>

Taskid is subtransaction numbering, transid is affairs numbering belonging to subtransaction, type is subtransaction type, wl charge capacity (1,000,000 instruction-level), cpu, mem, disk, band takies abstract resource (CPU, internal memory, disk and the bandwidth) quantity in virtual machine when being subtransaction execution.Although in the abstract resource of virtual machine, cpu resource can between multiple different subtransaction time-sharing multiplex, but quite a few video program belongs to computation-intensive affairs (such as encoding and decoding), quite large for cpu resource consumption, multiple computation-intensive affairs are run simultaneously on a CPU core, execution speed can be caused sharply to reduce, therefore need the quantity of the computation-intensive affairs on restriction processor.Therefore also cpu resource is abstracted into quantity to measure, a processor is abstracted into 4 cpu resources.Blockid is that affairs need data block to be processed numbering.Each element in limit set E represents a dependence between two subtransactions.E _{i, j}(e _{i, j}∈ E) represent subtransaction v _isubtransaction v _jforerunner, subtransaction v _jsubtransaction v _ifollow-up.

In conjunction with the feature of video data process, in the transaction scheduling and resource allocator model of the present invention's proposition, each back end has a dispatching process.These dispatching processes know the current state of the virtual machine of their place back end.And the dispatching process of different node can communicate with the dispatching process of client, transmit transaction scheduling and control information.After affairs are divided into subtransaction according to the video data blocking information tree Tr inquired by the dispatching process of client, according to binary relation set R, back end is performed cluster, clustering rule is as follows:

A _i∈ D _m, for any a _iif, a _j∈ D _m, a _i≠ a _j, then there is r _{i, j}∈ R

The all back end storing same data block m form a cluster set D _m, a _ifor cluster set D _man interior node.The dispatching process of client is from improving the angle of resource utilization and load balancing at D _minterior selection optimal node is executed the task.

If the dispatching process of back end receives affairs, then this affairs or insert the wait transaction queues of this node according to its priority, or the affairs seizing other low priority perform immediately.After affairs are finished, the dispatching process of back end just notifies the dispatching process of client, so that the dispatching process of client carries out the scheduling of subsequent transactions.

In cloud storage system, the number of nodes of the data block distribution of video data is huge, and each node therefore storing video data data block needs the load effectively reducing management node.In the transaction scheduling model that the present invention proposes, node has the ability of distributing own resource and transaction scheduling, has the ability of statistics own resource and load condition.

In the Dynamic Resource Allocation for Multimedia proposed in the present invention and the method for transaction scheduling, the dispatching process of client c is to cluster set D corresponding to data block m _minterior all back end (a ₁, a ₂, a ₃) send transaction request.D _minterior all back end according to internal resource and current loading condition, and utilize the dynamic factors extracted by history run status information to calculate affairs v _iat the estimation implementation effect of this node, and feed back to client dispatching process.Client dispatching process selects the node that can complete these affairs the earliest to perform affairs v according to feedback information _i.Thisly dominate scheduling process with client, back end dominates the mode of Resourse Distribute and dynamic feedback to the load balancing being conducive to whole cloud storage system.

1st step.The dispatching process of client c needs the blocking information tree Tr of video file to be processed to management node inquiry transaction.

2nd step.The dispatching process of client c is divided into multiple subtransaction according to the blocking information of video file these affairs, wherein each subtransaction process data block, and the result after division exports with the form of digraph GR.

3rd step.Judge the subtransaction whether existed in GR without forerunner, if there is no, then wait for.Otherwise, from GR, select a subtransaction all completed without forerunner or all forerunner's subtransactions, be assumed to be v _i.

4th step.Client c is by deblocking inforamtion tree Tr and subtransaction v _idata block numbering block in descriptor _icluster is performed to node, exports a node set D _blocki, D _blocki={ a ₁, a ₂, a _j..., a _n-1.Client c is to D _blockiin all nodes send transaction request, contain subtransaction v in transaction request _idescriptor.

5th step.Receive the node a of transaction request _jaccording to subtransaction v _icharge capacity (wl _i), transaction types and node a _jresource characteristics, estimator affairs v _iat node a _jestimation time ETM' _{i, j}.The present invention element ETM of transaction time matrix _{i, j}represent subtransaction v _iat node a _jestimation time.As subtransaction v _icharge capacity very large time, have larger deviation between estimated value with actual value.Therefore an adjustment factor fm is introduced _{k, j}come ETM' _{i, j}calibrate, obtain final TM _{i, j}.

ETM _i，j＝ETM' _i，j×fm _k，j

Wherein fm _{k, j}for kth class affairs are about node a _jadjustment factor.Adjustment factor fm _{k, j}reflect node a _jperform the historical characteristics of kth class affairs.Fm _{k, j}initial value be 1.

fm' _k，j＝Tact _i，j/ETM _i，j

fm _k，j＝(fm _k，j×λ+fm' _k，j×(1-λ))/2

Fm _{k, j}a updated value, node a _jafter often executing the subtransaction of a kth class, then calculate fm' _{k, j}, Tact _{k, j}for subtransaction k is at node a _jon the actual treatment time (comprise subtransaction start perform before or be preempted the rear wait resource ready time).If

| fm' _{k, j}-1|> Δ fm, then this fm' _{k, j}give up, Δ fm is a predefine threshold value, and λ is predefined weights, regulates fm respectively _{k, j}renewal speed and amplitude.If fm _{k, j}renewal amplitude is too little, then as node a _joverall load when changing, its value is easily aging, affects ETM _{i, j}the accuracy estimated.If fm _{k, j}renewal amplitude is too large, then its value is easily by the impact of indivedual local affairs, becomes unstable, affects ETM equally _{i, j}the accuracy estimated.

6th step.According to the feature carrying out preempting resources in transaction scheduling model according to priority, the present invention proposes resource queue's management method to calculate subtransaction v simultaneously _iat node a _jin resource ready time, be assumed to be ETA _{i, j}.With node a _jin the transaction queues version P, the node a that are being performed _jin waiting for transaction queues version W, the node a of resource _jmiddle current idle resource quantity ar _j; Subtransaction v _idescriptor for input; With subtransaction v _iat node a _jresource ready time ETA _{i, j}export.

1.ETA _{i, j}be initialized as current time.

2. affairs all in queue P are pushed into queue G successively.

3. affairs all in queue W are pushed queue G successively, suppose that the length of queue G is n.

4. according to subtransaction v _ipriority, by subtransaction v _iinsert queue G, suppose subtransaction v _iposition in queue G is i.

5. suppose subtransaction v _ithe resource needed is r _i, judge ar _jwhether be greater than r _i, in this way, then turn the 14th step.

6. suppose that k is subtransaction v _kposition in queue G.For each k=i+1 ..., n, if v _k∈ P, then lr _j=lr _j+ r _k, and v is removed from queue C _k.

7. the affairs performed in queue G are pushed queue G successively ₁, by queue G etc. pending affairs push queue G successively ₂.

8., according to the estimation time of each affairs and processing time (not comprising the stand-by period after being preempted), calculate queue G ₁in current residual processing time of each affairs, and carry out descending sort according to the current residual processing time of each affairs.The affairs that those current residual processing times can be allowed few complete as early as possible, releasing resource.

9. suppose queue G ₁current head of the queue affairs are vh ₁, upgrade ETA _{i, j}and ar _j:

ETA _{i, j}=ETA _{i, j}+ vh ₁the current residual processing time, ar _j=ar _j+ rh ₁.

10. eject queue G ₁head of the queue affairs vh ₁.

11. hypothesis queue G ₂current head of the queue affairs are vh ₂, judge ar _jwhether be greater than rh ₂, if not, then turn the 9th step.

12. by queue G ₂in head of the queue affairs vh ₂eject, and pushed queue G ₁afterbody, and upgrade arj:

ar _j＝ar _j-rh ₂

13. judge queue G ₂whether be empty, as no, then turn the 9th step.

14. export ETA _{i, j}, terminate.

7th step.Calculate affairs v _iat node a _jθ on earliest finish time _{i, j}.

θ _i，j＝ETA _i，j+ETM _i，j

ETM _{i, j}and ETA _{i, j}these two factors all can affect the accuracy of θ, and then directly affect the optimality of transaction scheduling and Resourse Distribute.For first factor ETM _{i, j}impact, by the adjustment factor fm in the 5th step _{k, j}come effectively to suppress.For second factor ETA _{i, j}impact, also need to introduce again an adjustment factor fd _jto in the computation process of θ.

Suppose affairs v _xfor node a _jthe historical transactions be finished.Executing affairs v _xafter, calculate node a at once _jupper up-to-date adjustment factor fd _j.

fdj＝α×(Tact_start _x-ETA _x，j)/ETA _x，j

Wherein Tact_start _xfor affairs v _xreal resource ready time (with respect to the resource ready time ETA that resource queue's management method is calculated _{x, j}), α is predefine constant.Adjustment factor fd _jthe meaning shown is: by the time quantity of time delay within the unit stand-by period.α, as a parameter, regulates adjustment factor fd _jθ is carried out to the amplitude of dynamic correcting.Due to the existence of preemption events, the adjustment factor fd calculated by the implementation effect of high priority affairs _jobviously less.The present invention expands fd for this reason _jdimension be bidimensional, i.e. fd _{j, l}, wherein j representation node, l represents priority.Affairs for different priorities use different adjustment factor fd _jcalculate affairs θ on earliest finish time.Introduce adjustment factor fd _jafter, the θ fd on earliest finish time that affairs are final _{i, j}computing method as follows.

θfd _i，j＝(1+fd _j)×ETA _i，j+ETM _i，j

8th step.Node a _jsubtransaction v is fed back to client c _iθ fd on earliest finish time _{i, j}.Client c is from D _blocki={ a ₁, a ₂, a _j..., a _n-1middle selection node a _xperform subtransaction v _i.Wherein θ fd _{i, x}meet:

θfd _i，x＝min(θfd _i，1，θfd _i，2，…，θfd _i，n-1)

Subtransaction v _inode a is assigned to by client c _xafter, insert the wait transaction queues of node ax according to its priority, or the affairs seizing other low priority perform immediately.

9th step.Client c judges whether GR is empty, and as being sky, then scheduling process terminates.If not for idling up to the 3rd step.

During in order to avoid a large amount of high priority affairs arrival node, in node, the stand-by period of low priority affairs in node waiting list constantly extends or is constantly preempted in execution queue, be absorbed in the state that the actual treatment time constantly extends, each node travels through the affairs in its waiting list with a predefine frequency, calculate its execution efficiency θ.

θ＝γ(Tacte _y/ETM _y，j)

Wherein Tacte _yfor affairs v _ythe current actual treatment time (comprising the time be preempted and in waiting list), γ is a predefine constant.By v _ypriority improve θ rank, simultaneously by Tacte _yreset, then according to affairs v _ypriority, again by affairs v _yinsert in the waiting list of node or seize low priority affairs to perform.

In order to meet the transactions requests of user, reduce failure recovery time, realize the data backup of lower expense, the present invention is further for data storage-type affairs, backed up by other server resources, when carrying out data backup operation, server priority selects expense lower and the internal resource that failure recovery time is shorter is used for storing data backup, when internal storage resources is not enough, select the resource of other servers for storing data backup.

By server ₁as the master server accepting user's business request, server ₂~ server _lother servers that data backup service can be provided.After master server receives user's request, reasonably can select the memory location of data backup according to storage overhead, communication overhead, select different server resources for storing data backup.

In master server, user can submit by cloud Platform communication interface the application used to Cloud Server, the state that inquiry transaction performs, and inspection institute stores the integrality of data; The situation of change of Resource Monitor charge of overseeing Cloud Server resource, collects the resource information of each server, for scheduler provides the current resource status of each server; Multiple data backups of user's business, according to the resource information of the transactions requests information of user and current each server, are dynamically stored in the resource of different servers according to predefine strategy, and scheduling result are fed back to cloud Platform communication interface by scheduler.When there being new user's business to arrive, data backup distributes by scheduler, to realize the backup services of lower expense.

After scheduling strategy calculates, cloud platform server is stored in the data backup of a user's business when master server provides _k(k=1,2 ..., L) after, then this backup will be stored in until arrive the storage time of user's application on corresponding cloud platform always, and the data backup be stored on other cloud platforms can not be write back on the cloud platform of master server in master server resource space idle again.

Each data backup cannot be stored on different server platforms by piecemeal again, and namely a backup can only be stored in the hardware resource of a server simultaneously.In order to avoid the redundancy of data, work as server _kreceive server ₁the request of a data back-up storage after, only need store a versions of data.

If server is current W user U ₁..., U _w, each user has multiple affairs N ₁..., N _w, the beginning processing time of each subtransaction can be different.With a four-tuple { t _ij, S _ij, T _ij, D _ijrepresent the resource request information of each affairs, wherein, t _ijrepresent a jth subtransaction of i-th user, i=1,2 ..., W; J=1,2 ..., N _i; S _ij, T _ijand D _ijrepresent affairs t respectively _ijstart time, storage time and memory capacity.

In the present invention, all storage backups can be stored to different server server according to overhead cost ₁..., server _lresource in.Assuming that the bandwidth that each server data stores is respectively BW ₁..., BW _l.

Preferably, the data backup expense cost of server mainly considers storage overhead and communication overhead 2 part of data:

$\cos t=\underset{i=1}{\overset{W}{\mathrm{\&Sigma;}}}\underset{j=1}{\overset{\mathrm{Ni}}{\mathrm{\&Sigma;}}}\underset{k=1}{\overset{L}{\mathrm{\&Sigma;}}}(T_{\mathrm{ijk}}\&times;D_{\mathrm{ij}}\&times;O_{\mathrm{ijk}}+D_{\mathrm{ij}}\&times;y_{\mathrm{ijk}})$

Wherein, T _ijkrepresent the time of a data back-up storage in a kth server resource of the affairs j of user i; D _ijrepresent the memory data output of a jth affairs of user i; O _ijkrepresent the versions of data number of a jth affairs of a kth server stores user i; y _ijkbe a Boolean type variable, work as server _kin (work as O when having the backup of the affairs j of user i _ijkwhen being more than or equal to 1), y _ijkequal 1, otherwise y _ijkbe 0.

In sum, the method that the present invention proposes improves the resource utilization of cloud plateform system and the effect of load balancing.

Obviously, it should be appreciated by those skilled in the art, above-mentioned of the present invention each module or each step can realize with general computing system, they can concentrate on single computing system, or be distributed on network that multiple computing system forms, alternatively, they can realize with the executable program code of computing system, thus, they can be stored and be performed by computing system within the storage system.Like this, the present invention is not restricted to any specific hardware and software combination.

Should be understood that, above-mentioned embodiment of the present invention only for exemplary illustration or explain principle of the present invention, and is not construed as limiting the invention.Therefore, any amendment made when without departing from the spirit and scope of the present invention, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.In addition, claims of the present invention be intended to contain fall into claims scope and border or this scope and border equivalents in whole change and modification.

Claims (3)

1. a cloud platform data disposal route, is characterized in that, comprising:

Video data file in cloud storage system is stored with the form of deblocking multi version and plans parallel processing transaction, with the mapping relations between data block and version for feature, cluster is performed to back end multiple in cloud platform, carrys out the unenforced affairs of dynamic dispatching according to the feedback information of executed affairs.

2. method according to claim 1, is characterized in that, described by the video data file in cloud storage system with the form of deblocking multi version store, comprise further:

The user's request of Video Applications is submitted to the client of cloud storage system with the form of mission description file, when affairs are submitted, client is divided into multiple subtransaction according to deblocking information affairs and carrys out executed in parallel, each subtransaction associates a data block, perform so that different subtransaction is distributed on different nodes, described deblocking information is described by tree Tr, Tr=(C, R), wherein C is a data element set, and each element represents a data block; R is a binary relation set, and whether the content that each element represents two data blocks is identical, the element c in described data element set C _ican describe with two tuple vectors: c _i=<block _i, a _i>, wherein block _irepresent the numbering of data block i, a _irepresent the numbering of data block i place node, the element r in described binary relation set R _{i, j}represent that data block j is the backup version of data block i.

3. method according to claim 2, it is characterized in that, each described back end has a dispatching process, the dispatching process of different node communicates with the dispatching process of client according to the current state of the virtual machine of place back end, transmit transaction scheduling and control information, after affairs are divided into subtransaction according to the video data blocking information tree Tr inquired by the dispatching process of client, according to binary relation set R, back end is performed cluster, clustering rule is:

For arbitrary node a _i, a _i∈ D _mif, a _j∈ D _mand a _i≠ a _j, then there is r _{i, j}∈ R

Wherein D _mfor storing the cluster set of all back end compositions of same data block m, the dispatching process of client is at D _minterior selection optimal node is executed the task;

If the dispatching process of back end receives affairs, then these affairs insert the wait transaction queues of this node according to its priority, or the affairs seizing other low priority perform immediately, after affairs are finished, the dispatching process of back end just notifies to make the dispatching process of client carry out the scheduling of subsequent transactions by the dispatching process of client.