CN103649948A - Key-value database data merging method and device - Google Patents

Key-value database data merging method and device Download PDF

Info

Publication number
CN103649948A
CN103649948A CN201280000718.XA CN201280000718A CN103649948A CN 103649948 A CN103649948 A CN 103649948A CN 201280000718 A CN201280000718 A CN 201280000718A CN 103649948 A CN103649948 A CN 103649948A
Authority
CN
China
Prior art keywords
task
combined
grade
load
urgency level
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201280000718.XA
Other languages
Chinese (zh)
Inventor
周帅锋
智伟
赵智峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of CN103649948A publication Critical patent/CN103649948A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present invention provides a key-value database data merging method and device, to control a data merging task, so as to prevent a heavier burden on the system incurred by concurrence of a large number of data merging tasks. The method comprises: obtaining a load level of a current system and an urgency level of a to-be-merged task; according to the load level of the current system and the urgency level of the to-be-merged task, matching a task-merging execution policy to determine whether the to-be-merged task is permitted to be executed; if it is determined that the to-be-merged task is permitted to be executed, sending the to-be-merged task to a task-merging execution module to execute task merging. The method provided by the present invention can control the data merging task according to the occupation of the system resources, so as to make the data merging task be automatically delayed when the load of the system is high, thereby reducing the impact of execution of task merging on service accessing, and reducing the burden on the system.

Description

Key-value database data merging method and device
The data merging method and engineering device technique field of key value database
The present invention relates to the data merging method and device of data processing field, more particularly to key value database.Background technology
Key assignments (Key- Value) database is the data base management system based on key-value data storage.Key assignments data storage technology is with the data model data storage of key-value pair, using the teaching of the invention it is possible to provide the function such as persistence mechanism and data syn-chronization.In some key value databases, data are according to key（Key) with sequential storage and to add pattern write-in disk, for example, BigTable and HBase are exactly this pattern.In such a mode, the file of file system is write, once after write-in, will not be changed.In this way, the data of different times write-in can form multiple storage files in file system.For the ease of accessing, keeping the orderly of data and clear up stale data in time; and improve the purposes such as the speed of data access; key value database can be after storage file reaches certain scale or continued for some time; start union operation; multiple files are merged, while carrying out some data scrubbing work.
In the key value database of some large-capacity distributings, sublist is to perform the base unit that data merge task, generally manages sublist by multiple sublist servers.For the ease of management, according to key（Key scope), many sublists are divided into by tables of data, and each sublist, which stores and manages key assignments, falls all data within the scope of it.
Data merging method is in a kind of key value database that prior art is provided：The base unit performed using sublist as scheduling, after the file number in sublist is up to or over the pre-defined upper limit, triggers merging task, the sublist is put into data and merges execution queue, merge thread by data and therefrom obtain sublist, perform merging, as shown in Figure 1.
Substantial amounts of system resource would generally be expended by merging a series of processes of task due to performing data, and all merging tasks unrestrictedly perform obviously meeting heavy system burden, influence the performance of data access business.The content of the invention
The embodiment of the present invention provides the data merging method and device of key value database, and merging task to data is controlled, to avoid mass data from merging task while carrying out heavy system burden.
The embodiment of the present invention provides a kind of data merging method of key value database, and methods described includes：Obtain The urgency level grade of the grade of load of current system and task to be combined；
According to the grade of load of the current system and the urgency level grade of task to be combined, Match merging appoints the grade of load of slightly current system higher, allow to perform urgency level higher grade task to be combined, when the grade of load of current system is saturation, do not allow to perform any task to be combined；
If it is determined that allowing to perform the task to be combined, then the task to be combined is sent to merging task execution module and perform task merging.
The embodiment of the present invention provides a kind of data merging device of key value database, and described device includes：Acquisition module, for obtaining the grade of load of current system and the urgency level grade of task to be combined；
Determining module, it is higher for the grade of load of current system that urgency level for the grade of load according to the current system and task to be combined etc. merges tasks carrying strategy, allow to perform urgency level higher grade task to be combined, when the grade of load of current system is saturation, do not allow to perform any task to be combined；Processing module, if determining to perform the task to be combined for the determining module, the task to be combined is sent to merging task execution module and performs task merging.
It was found from the embodiments of the present invention, due to task to be combined whether perform merge be the grade of load according to current system and task to be combined urgency level grade, Match merging tasks carrying strategy is determined, i.e. according to the system resource grade of load, it is allowed to perform the high task to be combined of urgency level.Therefore, method provided in an embodiment of the present invention can be according to system resource occupancy situation come control data merging task, allow data to merge task automatically delaying when system load is high, so as to reach that reduction performs task merging and access service is influenceed, mitigate system burden.
Brief description of the drawings
Technical scheme in order to illustrate the embodiments of the present invention more clearly, cylinder will be made to the required accompanying drawing used in prior art or embodiment description below singly to introduce, apparently, drawings in the following description are only some embodiments of the present invention, for those of ordinary skill in the art, without having to pay creative labor, other accompanying drawings can also be obtained such as these accompanying drawings.
Fig. 1 is data merging method schematic diagram in a kind of key value database that prior art is provided；
Fig. 2 is the data merging method schematic flow sheet of key value database provided in an embodiment of the present invention； Fig. 3 is the data merging device structural representation of key value database provided in an embodiment of the present invention；Fig. 4 is the data merging device structural representation for the key value database that another embodiment of the present invention is provided；Fig. 5 is the data merging device structural representation for the key value database that another embodiment of the present invention is provided；Fig. 6 a are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 6 b are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 7 a are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 7 b are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 7 c are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 7 d are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 8 a are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 8 b are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 8 c are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 8 d are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 9 a are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 9 b are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 9 c are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Fig. 9 d are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；Figure 10 a are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；
Figure 10 b are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；
Figure 10 c are the data merging device structural representations for the key value database that another embodiment of the present invention is provided；
Figure 10 d are the data merging device structural representations for the key value database that another embodiment of the present invention is provided.
Embodiment
The embodiment of the present invention provides the data merging method and device of key value database, merges task to dispatch data according to system resource occupancy situation. Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is only a part of embodiment of the invention, rather than whole embodiments.Based on the embodiment in the present invention, the every other embodiment that those of ordinary skill in the art are obtained under the premise of creative work is not made belongs to the scope of protection of the invention.
Accompanying drawing 2 is referred to, is the data merging method schematic flow sheet of key value database provided in an embodiment of the present invention, its executive agent can be the sublist server in key value database, mainly includes step S201, step S202 and step S203:
S201, obtains the grade of load of current system and the urgency level grade of task to be combined.
Because the load whether the merging task of key value database is performed with current system is relevant, for example, systematic comparison is busy and/or during higher system load, unrestrictedly perform key value database merging task can heavy system burden, influence the performance of data access business.In addition, whether the merging task of key value database performs relevant also with the urgency level of task to be combined.Therefore, in embodiments of the present invention, the grade of load of current system and the urgency level grade of task to be combined can be obtained first, wherein, it is several that the grade of load of current system includes but is not limited to low-load, middle load, high capacity and saturation etc., and the urgency level grade of task to be combined is including but not limited to non-emergent, common and urgent etc. several.
As the one embodiment for the grade of load for obtaining current system, the service condition of the resource of current system according to weight can be weighted summation, load imparting corresponding grade of the value according to obtained by the weighted sum to the current system.In embodiments of the present invention, central processing unit（Central Processing Unit, CPU), internal memory, network and disk etc. belong to the resource that current system is included, can be after the service condition of these resources be obtained, weight according to each resource is weighted summation, and the value obtained by weighted sum is the equal of the scoring that the load to current system is carried out, for example, if the load scoring of current system is represented using L,：Wherein, and ^ Fen Do represent CPU scorings weight, internal memory scoring weight, network scoring weight and disk scoring weight, and ucpu umT ^ are He Fen Do represent CPU service condition, internal memory service condition, Internet Use and disk service condition.If it is several that the load of system can be divided into low-load, middle load, high capacity and saturation etc. according to grade from low to high, then corresponding grade can be assigned successively by the load of current system from low to high according to the load scoring L of system, if for example, the load of system The L full marks that score are 100, then the load of system can be scored at 0 ~ 25,25-50,50 ~ 75 and 75 ~ 100 4 intervals, correspondingly assign low-load, middle load, high capacity and saturation by the grade of load of current system successively.
It is used as the one embodiment for the urgency level grade for obtaining task to be combined, the factor of the urgency level of the task to be combined can will be influenceed to carry out linear weighted function summation according to weight, urgency level corresponding grade of the value according to obtained by summing the linear weighted function to the task to be combined.In embodiments of the present invention, the factor of the urgency level of task to be combined is influenceed to include but is not limited to the file size in sublist, oldest data are exceeded the time limit time etc. in file number and sublist in sublist.Because different influence factors has the difference quantified, therefore the urgency level scoring of task to be combined can be calculated by the way of linear weighted function summation.In this manner, to each factor for the urgency level for influenceing task to be combined, percentage and weighted sum are gone out according to the maximum value calculation of each factor, it is as follows, if the urgency level scoring of task to be combined is used
ScoreiX^X^...^) ^^, shellfish1J：
Score{Xl,X2,...,Xk) = jwk{Xk I MAXXk) ^lQQ% formulas（2) wherein, ^ represents the factor of the urgency level of the influence task to be combined, ^ represents the weight of the factor of the urgency level of the influence task to be combined, and M X represent the maximum that the factor of the urgency level of the influence task to be combined can reach or set.If the urgency level grade of task to be combined can be divided into non-emergent, common and urgent etc. several from low to high, can then score ^^ (^ according to the urgency level of task to be combined, ..., corresponding grade ^) is assigned successively by the urgency level of task to be combined from low to high, if for example, the urgency level scoring ^ Ο Γ of task to be combined, Χ, ..., ^) full marks are 100, then the urgency level of task to be combined can be scored1¾^( 1, 2..., ^) at 0 ~ 30,30 ~ 60 and 60 ~ 100 3 intervals, it is correspondingly successively that the urgency level grade imparting of task to be combined is non-emergent, common and urgent.
According to formula（2) urgency level to task to be combined scores, consider be the general state of each factor under common scenarios when, but, under some scenes, although the score value of the factor of the urgency level of other influences task to be combined may be also than relatively low, but the factor of some urgency level for influenceing task to be combined already close to or when reaching maximum, the urgent degree entire effect that its execution to the task to be combined merges starts in non-linearly increasing.Due to formula（2) urgency level to task to be combined carries out use of scoring Linear weighted function summation, some influence task to be combined urgency level factor already close to or when reaching maximum, if still according to formula（2) urgency level to task to be combined scores, and its obtained appraisal result can not reflect that the task needs to be combined perform the urgency level merged strictly according to the facts, it is clear that unreasonable.
The embodiment of the present invention provides the methods of marking that a kind of individual event influence extreme value amplifies weighted sum, the urgency level of task to be combined is influenceed to reach or during close to maximum with Do, it non-linear can amplify the scoring to the urgency level of task to be combined, i.e., if currently influenceing the influence value of some factor in each factor of the urgency level of the task to be combined to reach or close to the maximum of the factor, each factor of the urgency level of the task to be combined will be then influenceed to carry out nonlinear weight summation according to weight, value according to obtained by summing the nonlinear weight assigns corresponding grade to the urgency level of the task to be combined.Specifically, each factor of the urgency level of the task to be combined will be influenceed, which to carry out nonlinear weight summation according to weight, to be included：The weight ^ of the factor of the urgency level of the task to be combined will be influenceed to be amplified to, using the new weight of the factor as the urgency level for influenceing the task to be combined, the factor of the urgency level of the task to be combined will be influenceed to carry out nonlinear weight summation according to the new weight, value according to obtained by summing the nonlinear weight is to the corresponding grade of urgency level of the task to be combined, i.e., urgency level scoring ^/^ of task to be combined：^,：^ ...,：^；！It is expressed as：
Score{Xl,X2,...,Xk) = jf{Ak)wk{Xkl ΜΑΧΧι) * 100% formula（3) wherein, ^ represents the factor of the urgency level of the influence task to be combined, and ^ represents the weight of the factor of the urgency level of the influence task to be combined, and M X represent the influence task to be combined, and it is weight magnification function, it can be expressed as：
f(Ak)=MAXXt /(MAXXt -Xk+AXk) formula（4) in above-mentioned expression formula, MAXXtRepresent the maximum of the factor influence value of the urgency level of the influence task to be combined, Δ ΧλIt is a minimum of the factor of the urgency level of the influence task to be combined, for adjustable type（4) so that formula（4) denominator is not 0.
It should be noted that above-mentioned according to formula（1) grade of load for the current system that the scoring calculated is obtained, and according to formula（2) or likes（3) the urgency level grade for the influence task to be combined that the scoring calculated is obtained, is merely illustrative, however it is not limited to above-mentioned formula（1), formula（2) or likes（3) side Formula.
S202, according to the grade of load of the current system and the urgency level grade of task to be combined, matching is in embodiments of the present invention, merge the urgency level grade that tasks carrying strategy is the grade of load according to system and task to be combined, pre-establish and be used to determine whether to perform a sets of plan of task merging, its principle exemplarily includes：The grade of load of current system is higher, allow to perform urgency level higher grade task to be combined, allow tasks carrying union operation to be combined to urgency level higher grade, when the grade of load of current system is saturation, do not allow to perform any task to be combined, i.e., do not allow the merging for performing any task to be combined.To merging tasks carrying strategy, it can also be read as：When the urgency level grade of task to be combined is determined, the grade of load of current system is higher, and the task to be combined allows the possibility for performing merging lower, on the contrary, the grade of load of current system is lower, and the task to be combined allows the possibility for performing merging higher；Or, when the grade of load of current system is determined, the urgency level higher grade of task to be combined（Task i.e. to be combined needs the degree for performing merging more urgent）, the task to be combined allows the possibility for performing merging higher, conversely, the urgency level lower grade of task to be combined（I.e. task to be combined needs to perform the degree merged and more relaxed Slow）, the task to be combined do not allow perform merge possibility it is higher；When the grade of load of current system is minimum, for example, when the grade of load of current system is low-load, can merge to the tasks carrying to be combined of any urgency level grade；In the grade of load highest of current system, for example, when the grade of load of current system is saturation, the tasks carrying to be combined merging to any urgency level grade is not allowed；In the urgency level grade highest of task to be combined, for example, when the urgency level grade of task to be combined is urgent, unless the grade of load of current system is highest, for example, the grade of load of current system is saturation, otherwise, all allows to merge the urgency level grade for urgent tasks carrying to be combined.
According to as previously described, the grade of load of current system includes but is not limited to several embodiment such as urgency level grade is including but not limited to non-emergent, common and urgent of several and to be combined tasks such as low-load, middle load, high capacity and saturation, and merging tasks carrying strategy provided in an embodiment of the present invention can be as shown in table 1 below：
Table 1
In table 1 above, " grade of load " is that the cylinder of " grade of load of current system " in previous embodiment claims, including " low-load ", " middle to load ", " high capacity " and " saturation ", " urgency level grade " is that the cylinder of " the urgency level grade of task to be combined " in previous embodiment claims, including " non-emergent ", " common " and " urgent ".The project of " grade of load " and " urgency level grade " infall is the merging for indicating whether to perform task to be combined, for example, " grade of load " is " low-load " and " urgency level grade " when being " non-emergent ", then can perform the merging of task to be combined.The merging tasks carrying strategy general principle that above-mentioned table 1 is reflected is, it is non-emergent task to be combined for urgency level grade, when the grade of load of system is middle load or the middle load above, do not allow to perform the business to be combined merging, but the task to be combined is returned into queue and waited；It is common task to be combined for urgency level grade, when the grade of load of system is more than high capacity or high capacity, does not allow to perform the business to be combined merging, but the task to be combined is returned into queue and waited；It is urgent task to be combined for urgency level grade, as long as the grade of load of the grade of load of system and unsaturation, i.e. current system is low-load, middle load or high capacity, then allows to perform merging to the business to be combined.According to the grade of load of the current system and the urgency level grade of task to be combined, Match merging tasks carrying strategy is included with determining whether to perform the task to be combined：If the urgency level grade for matching the task to be combined is urgent, when the grade of load of the current system is low-load, middle load or high capacity, then determine to allow to perform the task to be combined, that is, determine to allow to merge the tasks carrying to be combined；If the urgency level grade for matching the task to be combined is non-emergent, when the grade of load of the current system is middle load or the middle load above, then determine not allow to perform the task to be combined, that is, determine not allowing to merge the tasks carrying to be combined；If The urgency level grade for being fitted on the task to be combined is common, when the grade of load of the current system is more than high capacity or high capacity, it is determined that do not allow to perform the task to be combined, that is, determines not allowing to merge the tasks carrying to be combined.
For the merging tasks carrying strategy of the example of table 1, if the grade of load for the current system that step S201 is obtained is " low-load ", then match the merging tasks carrying strategy of the example of table 1, no matter the urgency level grade for the task to be combined that step S201 is obtained is " non-emergent ", " common " or " urgent ", all determines that the merging of the task to be combined can be performed；If the grade of load for the current system that step S201 is obtained is " middle to load ", then match the merging tasks carrying strategy of the example of table 1, when the urgency level grade of the step S201 tasks to be combined obtained is " common " or " urgent ", also determine that the merging of the task to be combined can be performed；If the grade of load for the current system that step S201 is obtained is " high capacity ", then match the merging tasks carrying strategy of the example of table 1, only when the urgency level grade of the step S201 tasks to be combined obtained is " urgent ", just determine that the merging of the task to be combined can be performed；If the grade of load for the current system that step S201 is obtained is " saturation ", then match the merging tasks carrying strategy of the example of table 1, no matter the urgency level grade for the task to be combined that step S201 is obtained is " non-emergent " " common " or " urgent ", all determines that the merging of the task to be combined cannot be performed；If the grade of load for the current system that step S201 is obtained is " high capacity ", then match the merging tasks carrying strategy of the example of table 1, when the urgency level grade of the step S201 tasks to be combined obtained is " non-emergent " or " common ", also determine that the merging of the task to be combined cannot be performed；If the grade of load for the current system that step S201 is obtained is " middle to load ", then match the merging tasks carrying strategy of the example of table 1, when the urgency level grade of the step S201 tasks to be combined obtained is " non-emergent ", also determine that the merging of the task to be combined cannot be performed.
S203, however, it is determined that allow to perform the task to be combined, then send the task to be combined to merging task execution module and perform task merging.
In embodiments of the present invention, it is the functional unit for performing task merging to merge task execution module, there may be one or more such functional units in sublist server.When it is that can perform task merging to merge task execution module to have the free time, then task actively is asked for scheduler, or active reporting itself is in idle condition, waits scheduler to distribute task.Therefore, according to step S201 and step S202 ,-denier determines that the task to be combined can be performed, that is, determines that the tasks carrying to be combined can be merged, then by institute State task to be combined to send to task execution module is merged, task merging is performed by merging task execution module.If it is determined that the task to be combined cannot be performed, that is, determine that the tasks carrying to be combined cannot be merged, then the task to be combined is returned to waiting for into queue, wait suitable opportunity to take out to perform task merging again from waiting list.
In embodiments of the present invention, with the grade residing for the load of current system, it is possible within one long time, sublist server should not all perform task merging, if for example, being supported in one section of longer time for current system can be in " saturation " this grade, according to the merging tasks carrying strategy of the example of table 1, no matter the urgency level grade of task to be combined is " non-emergent ", " common " or " urgent ", cannot all perform the merging of the task to be combined.If within a period of time of above-mentioned example, task still continually from can not perform task merging according to merging tasks carrying strategy before waiting list taking-up and return to waiting for queue, it is clear that be a kind of behavior of waste of resource.
Therefore, in order to avoid sublist server should not all perform task merging and still continually take out task to be combined from waiting list and go to perform task merging within one long time, in embodiments of the present invention, after the task to be combined that merging tasks carrying strategy according to the example of table 1 in step S201 and step S202 can not be performed task merging and be returned to waiting in queue returns to waiting for queue, the task to be combined can be freezed, so that the task to be combined is in the freeze-off time of setting can not take out the state of the waiting list.The size set as freeze-off time, can be set by the user, the present invention is limited this Bu be Te Do according to the length of the grade of load of current system and/or the urgency level grade expected duration of the task to be combined.
In embodiments of the present invention, after the task to be combined is freezed, a timer can also be started, timing is carried out to the freeze-off time of the setting, so that after the freeze-off time of the setting terminates, the task to be combined that merging tasks carrying strategy according to the example of table 1 in step S201 and step S202 can not be performed task merging and be returned to waiting in queue is changed to that the state of waiting list can be taken out, so as to reappraise whether the task to be combined can perform merging.
Because freeze-off time is the length of the expected duration based on grade residing for the load of current system and is set, therefore, expected duration of the setting more than grade residing for the load of current system of freeze-off time can not be excluded.If for example, the urgency level grade of the grade of load of current system or/and the task to be combined changes, so that according to the merging tasks carrying strategy of the example of table 1, to before according to the conjunction of the example of table 1 It is also a kind of waste of resource for system if mechanically waiting the freeze-off time of setting to terminate just to perform task merging now and tasks carrying strategy can not perform the task to be combined of merging can re-execute merging.
Therefore, in embodiments of the present invention, the change occurred for the grade of load of current system, i.e., if the grade of load of the current system drops to " high capacity ", " middle to load " or " low-load " by " saturation ", the task to be combined that the merging tasks carrying strategy according to the example of table 1 in step S201 and step S202 can not be performed task merging and be returned to waiting in queue is changed to that the state of waiting list can be taken out；Or, if the grade of load of the current system is dropped to " middle to load " or " low-load " by " high capacity ", the task to be combined that the merging tasks carrying strategy according to the example of table 1 in step S201 and step S202 can not be performed task merging and be returned to waiting in queue is changed to that the state of waiting list can be taken out；Or, if the grade of load of the current system drops to " low-load " by " middle to load ", the task to be combined that the merging tasks carrying strategy according to the example of table 1 in step S201 and step S202 can not be performed task merging and be returned to waiting in queue is changed to that the state of waiting list can be taken out.
In an alternative embodiment of the invention, the change that urgency level grade for task to be combined occurs, the urgency level grade that can not be performed task merging according to the merging tasks carrying strategy of the example of table 1 even in step S201 and step S202 and return to waiting for the task to be combined in queue is risen to " common " or " urgent " by " non-emergent ", then by the task to be combined is changed to that the state of waiting list can be taken out；Or, the urgency level grade that the task to be combined in queue is returned to waiting for if can not perform task merging according to the merging tasks carrying strategy of the example of table 1 in step S201 and step S202 rises to " urgent " by " common ", and the task to be combined is changed to that the state of waiting list can be taken out.
Terminate the another embodiment of freeze-off time in advance as the present invention, when all tasks to be combined are freezed and merge task execution module in idle condition in waiting list, if the urgency level grade that can not be performed task merging in step S201 and step S202 and return to waiting for the urgency level grade task more to be combined than other of the task to be combined in queue according to the merging tasks carrying strategy of the example of table 1 is high, then the task to be combined that can not be performed task merging according to the merging tasks carrying strategy of the example of table 1 in step S201 and step S202 and return to waiting in queue is changed to that the state of the waiting list can be taken out.
The data merging method of the key value database provided from the embodiments of the present invention is understood, because whether task to be combined performs the urgency level grade that merging is the grade of load according to current system and task to be combined, Match merging tasks carrying strategy is determined.Therefore, load or the state of system are not considered with prior art（For example, systematic comparison is busy and/or system load is higher）And the merging for still unrestrictedly performing task to be combined is compared, method provided in an embodiment of the present invention can dispatch data according to system resource occupancy situation and merge task, data are allowed to merge task automatically delaying when system load is high, performed automatically in system low-load, so as to reach reduction perform task merging access service influence while, can allow again task to be combined suitable opportunity automatically, perform merging in time.
Accompanying drawing 3 is referred to, is the data merging device structural representation of key value database provided in an embodiment of the present invention.For convenience of description, it illustrate only the part related to the embodiment of the present invention.The data merging device of the key value database of the example of accompanying drawing 3 includes acquisition module 301, determining module 302 and processing module 303, wherein：
Acquisition module 301, for obtaining the grade of load of current system and the urgency level grade of task to be combined.
Because the load whether the merging task of key value database is performed with current system is relevant, for example, systematic comparison is busy and/or during higher system load, unrestrictedly perform key value database merging task can heavy system burden, influence the performance of data access business.In addition, whether the merging task of key value database performs relevant also with the urgency level of task to be combined.Therefore, in the present embodiment, the grade of load of current system and the urgency level grade of task to be combined can be obtained by grade acquisition module 301 first, wherein, it is several that the grade of load of current system includes but is not limited to low-load, middle load, high capacity and saturation etc., and the urgency level grade of task to be combined is including but not limited to non-emergent, common and urgent etc. several.
Determining module 302, urgent journey for the grade of load according to the current system and task to be combined is in the present embodiment, merge the urgency level grade that tasks carrying strategy is the grade of load according to system and task to be combined, pre-establish and be used to determine whether to perform a sets of plan of task merging, its principle exemplarily includes：The grade of load of current system is higher, allow to perform the higher task to be combined of urgency level, tasks carrying to be combined to urgency level higher grade is allowed to merge, when the grade of load of current system is saturation, do not allow to perform any task to be combined, i.e., do not allow the merging for performing any task to be combined.In embodiments of the present invention, to merging tasks carrying strategy, it can also be read as：When the urgency level grade of task to be combined is determined, the grade of load of current system is higher, and the task to be combined allows to perform conjunction And possibility it is lower, conversely, the grade of load of current system is lower, the task to be combined allows to perform the possibility merged higher；Or, when the grade of load of current system is determined, the urgency level higher grade of task to be combined（Task i.e. to be combined needs the degree for performing merging more urgent), the task to be combined allows the possibility for performing merging higher, conversely, the urgency level lower grade of task to be combined（I.e. task to be combined needs to perform the degree merged and more relaxed Slow）, the task to be combined do not allow perform merge possibility it is higher；When the grade of load of current system is minimum, for example, when the grade of load of current system is low-load, can merge to the tasks carrying to be combined of any urgency level grade；In the grade of load highest of current system, for example, when the grade of load of current system is saturation, the tasks carrying to be combined merging to any urgency level grade is not allowed；In the urgency level grade highest of task to be combined, for example, when the urgency level grade of task to be combined is urgent, unless the grade of load of current system is highest, for example, the grade of load of current system is saturation, otherwise, all allows to merge the urgency level grade for urgent tasks carrying to be combined.
According to as previously described, the grade of load of current system includes but is not limited to several embodiment such as urgency level grade is including but not limited to non-emergent, common and urgent of several and to be combined tasks such as low-load, middle load, high capacity and saturation, and the merging tasks carrying strategy that the present embodiment is provided can be as shown in foregoing table 1.
Processing module 303, if determining to perform the task to be combined for the determining module 302, the task to be combined is sent to merging task execution module and performs task merging.
In the present embodiment, it is the functional unit for performing task merging to merge task execution module, there may be one or more such functional units in sublist server.When it is that can perform task merging to merge task execution module to have the free time, then task actively is asked for scheduler, or active reporting itself is in idle condition, waits scheduler to distribute task.Therefore, once it is determined that module 302 determines that the task to be combined can be performed, determine that the tasks carrying to be combined can be merged, then processing module 303 sends the task to be combined to task execution module is merged, and task merging is performed by merging task execution module.If it is determined that module 302 determines that the task to be combined cannot be performed, determine that the tasks carrying to be combined cannot be merged, then processing module 303 can also return to waiting for the task to be combined queue, wait suitable opportunity to take out to perform task merging again from waiting list.
It should be noted that in the embodiment of the data merging device of above key value database, the division of each functional module is merely illustrative of, and can be as needed in practical application, such as the configuration requirement of corresponding hardware Or the convenient consideration of the realization of software, and above-mentioned functions are distributed and completed by different functional modules, will the internal structure of data merging device of the key value database be divided into different functional modules, to complete all or part of function described above.And, in practical application, corresponding functional module in the present embodiment can be realized by corresponding hardware, corresponding software can also be performed by corresponding hardware to complete, for example, foregoing grade acquisition module, can be the hardware with the urgency level grade for performing the foregoing grade of load for obtaining current system and task to be combined, such as grade getter or corresponding computer program is able to carry out so as to processor or other hardware devices completing foregoing function；For another example foregoing determining module, can have to perform the foregoing grade of load and the urgent journey part of task to be combined according to the current system, such as determiner or corresponding computer program is able to carry out so as to processor or other hardware devices completing foregoing function（Each embodiment that this specification is provided can all apply foregoing description principle）.
The acquisition module 301 of the example of accompanying drawing 3 can include the data merging device of first acquisition unit 401, the as shown in Figure 4 key value database that another embodiment of the present invention is provided.The service condition that first acquisition unit 401 is used for the resource for including current system is weighted summation according to weight, load of the value to the current system according to obtained by the weighted sum assigns corresponding grade, wherein, the service condition of the resource of the system includes any combination of CPU service condition, internal memory service condition, Internet Use and disk service condition, and the grade of load of the current system includes low-load, middle load, high capacity and saturation.
In the data merging device of the key value database of the example of accompanying drawing 4, central processing unit（Central Processing Unit, CPU), internal memory, network and disk etc. belong to the resource that current system is included.First acquisition unit 401 can be after the service condition of these resources be obtained, weight according to each resource is weighted summation, value obtained by weighted sum is the equal of the scoring that the load to current system is carried out, for example, if the load scoring of current system is represented using L, L expression formula such as previously described formula（1) shown in.If the load of system can be divided into low-load from low to high according to grade, middle load, high capacity and saturation etc. are several, then the load of current system can be assigned corresponding grade successively by first acquisition unit 401 from low to high according to the load scoring L of system, for example, if the load scoring L full marks of system are 100, the load of system is then scored into L 0 ~ 25, 25-50, during 50 ~ 75 and 75 ~ 100 4 intervals, the grade of load of current system is correspondingly assigned low-load by first acquisition unit 401 successively, middle load, high capacity and saturation. It should be noted that, in the data merging device of the key value database of the example of accompanying drawing 4, its acquisition module 301 can also include other functional modules or unit, and the urgency level grade for obtaining task to be combined, its acquisition modes are not limited in the device of the example of accompanying drawing 4.
The grade acquisition module 301 of the example of accompanying drawing 3 can also include the data merging device of second acquisition unit 501, the as shown in Figure 5 key value database that another embodiment of the present invention is provided.The factor that second acquisition unit 501 is used for the urgency level by the task to be combined is influenceed carries out linear weighted function summation according to weight, value according to obtained by summing the linear weighted function assigns corresponding grade to the urgency level of the task to be combined, wherein, each factor of the urgency level of the influence task to be combined includes any combination of the time of exceeding the time limit of oldest data in file number in the file size in sublist, sublist and sublist, and the urgency level grade of the task to be combined is including common, urgent and necessary.
In the data merging device of the key value database of the example of accompanying drawing 5, the factor of the urgency level of task to be combined is influenceed to include but is not limited to the file size in sublist, oldest data are exceeded the time limit time etc. in file number and sublist in sublist.Because different influence factors has the difference quantified, therefore, the urgency level that second acquisition unit 501 can calculate task to be combined by the way of linear weighted function summation scores.In this manner, to each factor for the urgency level for influenceing task to be combined, percentage and weighted sum are gone out according to the maximum value calculation of each factor, if the urgency level scoring Srar previously described formula of task to be combined（2) represent, the urgency level grade of task to be combined can be divided into from low to high it is common, urgent and must etc. it is several, then second acquisition unit 501 can score according to the urgency level of task to be combined
, corresponding grade is assigned successively by the urgency level of task to be combined from low to high, if for example, the urgency level scoring of task to be combined ) full marks are 100, then the urgency level of task to be combined is scored. (1, 2..., ^) at 0 ~ 30,30 ~ 60 and 60 ~ 100 3 intervals, second acquisition unit 501 is correspondingly common, urgent and necessary by the urgency level grade imparting of task to be combined successively.
It should be noted that, in the data merging device of the key value database of the example of accompanying drawing 5, its acquisition module 301 can also include other functional modules or unit, and the grade of load for obtaining current system, its acquisition modes are not limited in the device of the example of accompanying drawing 5.
According to formula（2) urgency level to task to be combined scores, it is considered to be the general state of each factor under common scenarios when, however, under some scenes, although other influences task to be combined is urgent The score value of the factor of degree may also than relatively low, but some urgency level for influenceing task to be combined factor already close to or when reaching maximum, the urgent degree entire effect that its execution to the task to be combined merges starts in non-linearly increasing.Due to formula（2) urgency level of task to be combined is carried out scoring use linear weighted function summation the urgency level of some influence task to be combined factor already close to or when reaching maximum, second acquisition unit 501 is still according to formula（2) urgency level to task to be combined scores, and its obtained appraisal result can not reflect the urgency level of the task to be combined strictly according to the facts, it is clear that unreasonable.It is necessary that Do influences the urgency level of task to be combined to reach or during close to maximum, it non-linear can amplify the scoring to the urgency level of task to be combined, i.e., if currently influenceing the influence value of factor in each factor of the urgency level of the task to be combined to reach or close to the maximum of the factor, then the acquisition module 301 of the example of accompanying drawing 3 can also include the data merging device of the 3rd acquiring unit 601, the as shown in fig. 6 key value database that another embodiment of the present invention is provided.
3rd acquiring unit 601, each factor for the urgency level by the task to be combined is influenceed carries out nonlinear weight summation according to weight, value according to obtained by the weighted sum assigns corresponding grade to the urgency level of the task to be combined, each factor of the urgency level of the influence task to be combined includes any combination of the time of exceeding the time limit of oldest data in file number in the file size in sublist, sublist and sublist, and the urgency level grade of the task to be combined is including non-emergent, common and urgent.
It should be noted that, in the data merging device of the key value database of accompanying drawing 6a examples, its acquisition module 301 can also include other functional modules or unit, and the grade of load for obtaining current system, its acquisition modes are not limited in the device of accompanying drawing 6a examples.
3rd acquiring unit 601 of accompanying drawing 6a examples may further include amplifying unit 6011 and sum unit
6012, as shown in fig. 6b another embodiment of the present invention provide key value database data merging device, wherein：
Amplifying unit 6011, it is weight magnification function that the weight ^ for the factor of the urgency level by the task to be combined is influenceed, which is amplified to described,.
Sum unit 6012, for using the new weight of the factor as the urgency level for influenceing the task to be combined, the factor of the urgency level of the task to be combined will be influenceed to be weighted summation according to the new weight, the corresponding grade of urgency level of value according to obtained by the weighted sum to the task to be combined, that is, sum unit 6012 is according to formula（3) urgency level to the task to be combined assigns phase The grade answered, including common, urgent and necessary etc..
Accompanying drawing 4 to the determining module 302 of accompanying drawing 6b examples can include the first determining unit 701, the second determining unit 702 and the 3rd determining unit 703, the data merging device for the key value database that another embodiment of the present invention as shown in accompanying drawing 7a to accompanying drawing 7d is provided, wherein：
First determining unit 701, if the urgency level grade for matching the task to be combined is urgent, when the grade of load of the current system is low-load, middle load or high capacity, it is determined that allow the merging for performing the task to be combined.
I.e., if the urgency level grade that acquisition module 301 obtains the task to be combined is " urgent ", then match the merging tasks carrying strategy of the example of table 1, the grade of load for the current system that acquisition module 301 is obtained is " low-load ", first determining unit 701 determines to allow to perform the task to be combined, determine to allow to merge the tasks carrying to be combined, or, the grade of load for the current system that acquisition module 301 is obtained is " middle to load ", first determining unit 701 determines to allow to perform the task to be combined, determine to allow to merge the tasks carrying to be combined, or, the grade of load for the current system that acquisition module 301 is obtained is " high capacity ", first determining unit 701 determines to allow to perform the task to be combined, determine to allow to merge the tasks carrying to be combined.
Second determining unit 702, if the urgency level grade for matching the task to be combined is non-emergent, when the grade of load of the current system is middle load or the middle load above, then determine not allow to perform the task to be combined, that is, determine not allowing to merge the tasks carrying to be combined.
I.e., if the urgency level grade that acquisition module 301 obtains the task to be combined is " non-emergent ", then match the merging tasks carrying strategy of the example of table 1, the grade of load for the current system that acquisition module 301 is obtained is " middle to load ", second determining unit 702 determines not allow to perform the task to be combined, determine not allowing to merge the tasks carrying to be combined, or, the grade of load for the current system that acquisition module 301 is obtained is " high capacity ", second determining unit 702 determines not allow to perform the task to be combined, determine not allowing to merge the tasks carrying to be combined, or, the grade of load for the current system that acquisition module 301 is obtained is " saturation ", second determining unit 702 determines not allow to perform the task to be combined, determine not allowing to merge the tasks carrying to be combined.
3rd determining unit 703, if the urgency level grade for matching the task to be combined is common, when the grade of load of the current system is more than high capacity or high capacity, it is determined that do not allow execution described Task to be combined.
I.e., if the urgency level grade for the task to be combined that acquisition module 301 is obtained is " common ", then match the merging tasks carrying strategy of the example of table 1, when the grade of load of the current system obtained in acquisition module 301 is " high capacity " or " saturation ", 3rd determining unit 703 determines not allow to perform the task to be combined, that is, determines not allowing to merge the tasks carrying to be combined.
Accompanying drawing 4 to the data merging device of the key value database of accompanying drawing 6b examples can freeze module 801 including task, the data merging device for the key value database that another embodiment of the present invention as shown in accompanying drawing 8a to accompanying drawing 8d is provided.Task, which freezes module 801, to be used to freeze the task to be combined, so that the task to be combined is in the freeze-off time of setting can not take out the state of the waiting list.
With the grade residing for the load of current system, it is possible to which within one long time, sublist server should not all perform task merging, if for example, being supported in one section of longer time for current system can be in
" saturation " this grade, then according to the merging tasks carrying strategy of the example of table 1, no matter the urgency level grade of task to be combined is " common ", " urgent " or " necessary ", cannot all perform the merging of the task to be combined.If within a period of time of above-mentioned example, task still continually from can not perform task merging according to merging tasks carrying strategy before waiting list taking-up and return to waiting for queue, it is clear that be a kind of behavior of waste of resource.Therefore, in order to avoid sublist server should not all perform task merging and still continually take out task to be combined from waiting list and go to perform task merging within one long time, in accompanying drawing 8a to the key value database of accompanying drawing 8d examples data merging device, merging tasks carrying strategy according to the example of table 1 can not be performed into task merging and return to waiting for the task to be combined in queue and return to waiting for after queue, task, which freezes module 801, to freeze the task to be combined, so that the task to be combined is in the freeze-off time of setting can not take out the state of the waiting list.The size set as freeze-off time, can be set by the user, the present invention is limited this Bu be Te Do according to the length of the grade of load of current system and/or the urgency level grade expected duration of the task to be combined.
The data merging device for the key value database that accompanying drawing 8a can be provided to the data merging device of the key value database of accompanying drawing 8d examples including the first defrosting module 901, another embodiment of the present invention as shown in accompanying drawing 9a to accompanying drawing 9d.First defrosting module 901 is used to start freeze-off time progress timing of the timer to the setting, so as to after the freeze-off time of the setting terminates, the task to be combined is changed to that the state of the waiting list can be taken out. Because freeze-off time is the length of the expected duration based on grade residing for the load of current system and is set, therefore, expected duration of the setting more than grade residing for the load of current system of freeze-off time can not be excluded.For example, if the urgency level grade of the grade of load of current system or/and the task to be combined changes, so as to according to the merging tasks carrying strategy of the example of table 1, merging can be re-executed to the task to be combined that can not perform merging according to the merging tasks carrying strategy of the example of table 1 before, now, it is also a kind of waste of resource for system if mechanically waiting the freeze-off time of setting to terminate just to perform task merging.Therefore, accompanying drawing 8a can also include the second defrosting module 1001, the 3rd defrosting module 1002 or the 4th defrosting module 1003 to the data merging device of the key value database of accompanying drawing 8d examples, the data merging device for the key value database that another embodiment of the present invention as shown in accompanying drawing 10a to accompanying drawing 10d is provided, wherein：
Second defrosting module 1001, if the grade of load for the current system drops to high capacity, middle load or low-load by saturation, middle load or low-load are dropped to by high capacity or drops to low-load by middle load, the task to be combined is changed to that the state of the waiting list can be taken out.
3rd defrosting module 1002, if the urgency level grade for the task to be combined is risen to common or urgent or risen to urgent by common by non-emergent, the task to be combined is changed to that the state of the waiting list can be taken out.
4th defrosting module 1003, for in the waiting list all tasks to be combined freeze and merge task execution module be in idle condition when, if the urgency level grade of the urgency level grade task more to be combined than other of the task to be combined is high, the task to be combined is changed to that the state of the waiting list can be taken out.
It should be noted that, the contents such as information exchange, implementation procedure between each module/unit of said apparatus, due to being based on same design with the inventive method embodiment, its technique effect brought is identical with the inventive method embodiment, particular content can be found in the narration in the inventive method embodiment, and here is omitted.
One of ordinary skill in the art will appreciate that all or part of step in the various methods of above-described embodiment can be by program to instruct the hardware of correlation to complete, such as following various methods it is one or more or whole：
Obtain the grade of load of current system and the urgency level grade of task to be combined；
According to the grade of load of the current system and the urgency level grade of task to be combined, Match merging is appointed The grade of load of slightly current system is higher, it is allowed to perform urgency level higher grade task to be combined, when the grade of load of current system is saturation, does not allow to perform any task to be combined；
If it is determined that performing the task to be combined, then the task to be combined is sent to merging task execution module and perform merging task.
One of ordinary skill in the art will appreciate that all or part of step in the various methods of above-described embodiment can be by program to instruct the hardware of correlation to complete, the program can be stored in a computer-readable recording medium, and storage medium can include：Read-only storage（ROM, Read Only Memory), random access memory（RAM, Random Access Memory), disk or CD etc..
The explanation for having carried out detailed embodiment to the data merging method and device of key value database provided in an embodiment of the present invention above is only intended to the method and its core concept for helping to understand the present invention；Simultaneously for those of ordinary skill in the art, according to the thought of the present invention, it will change in specific embodiments and applications, in summary, this specification content should not be construed as limiting the invention.

Claims (16)

  1. Claim
    1st, the data merging method of a kind of key value database, it is characterised in that methods described includes：Obtain the grade of load of current system and the urgency level grade of task to be combined；
    According to the grade of load of the current system and the urgency level grade of task to be combined, Match merging appoints the grade of load of slightly current system higher, allow to perform urgency level higher grade task to be combined, when the grade of load of current system is saturation, do not allow to perform any task to be combined；
    If it is determined that allowing to perform the task to be combined, then the task to be combined is sent to merging task execution module and perform task merging.
    2nd, the method as described in claim 1, it is characterised in that the grade of load of the acquisition current system includes：
    The service condition of the resource of current system is weighted summation according to weight, load of the value to the current system according to obtained by the weighted sum assigns corresponding grade, the service condition of the resource of the system includes any combination of CPU service condition, internal memory service condition, Internet Use and disk service condition, and the grade of load of the current system includes low-load, middle load, high capacity and saturation successively according to low to high.
    3rd, the method as described in claim 1, it is characterised in that the urgency level grade of the acquisition task to be combined includes：
    Each factor of the urgency level of the task to be combined will be influenceed to carry out linear weighted function summation according to weight, value according to obtained by summing the linear weighted function assigns corresponding grade to the urgency level of the task to be combined, each factor of the urgency level of the influence task to be combined includes any combination of the time of exceeding the time limit of oldest data in file number in the file size in sublist, sublist and sublist, and the urgency level grade of the task to be combined is including non-emergent, common and urgent.
    4th, the method as described in claim 1, it is characterized in that, if currently influenceing the influence value of the factor in each factor of the urgency level of the task to be combined to reach or that close to the maximum of the factor urgency level grade for obtaining task to be combined includes：
    Each factor of the urgency level of the task to be combined will be influenceed to carry out nonlinear weight summation according to weight, the value according to obtained by summing the nonlinear weight assigns corresponding to the urgency level of the task to be combined Grade, each factor of the urgency level of the influence task to be combined includes any combination of the time of exceeding the time limit of oldest data in file number in the file size in sublist, sublist and sublist, and the urgency level grade of the task to be combined is including non-emergent, common and urgent.
    5th, method as claimed in claim 4, it is characterised in that each factor of the urgency level that will influence the task to be combined carries out nonlinear weight summation according to weight to be included：
    The weight ^ of the factor of the urgency level of the task to be combined will be influenceed to be amplified to, described is weight magnification function；
    Using the new weight of the factor as the urgency level for influenceing the task to be combined, the factor of the urgency level of the task to be combined will be influenceed to carry out nonlinear weight summation according to the new weight, value according to obtained by summing the nonlinear weight assigns corresponding grade to the urgency level of the task to be combined, and the urgency level grade of the task to be combined is including non-emergent, common and urgent.
    6th, the method as described in claim 2 to 5 any one, it is characterized in that, the grade of load and the urgency level grade of task to be combined according to the current system, Match merging tasks carrying strategy is included with determining whether to perform the task to be combined：
    If the urgency level grade for matching the task to be combined is urgent, when the grade of load of the current system is low-load, middle load or high capacity, it is determined that allow to perform the task to be combined；
    If the urgency level grade for matching the task to be combined is non-emergent, when the grade of load of the current system is middle load or the middle load above, it is determined that do not allow to perform the task to be combined；
    If the urgency level grade for matching the task to be combined is common, when the grade of load of the current system is more than high capacity or high capacity, it is determined that do not allow to perform the task to be combined.
    7th, the method as described in claim 2 to 5 any one, it is characterised in that described to return to waiting for further comprising after queue by the task to be combined：
    The task to be combined is freezed, so that the task to be combined is in the freeze-off time of setting can not take out the state of the waiting list.
    8th, method as claimed in claim 7, it is characterised in that it is described the task to be combined is freezed after further comprise：
    Start timer and timing is carried out to the freeze-off time of the setting, so as to after the freeze-off time of the setting terminates, the task to be combined is changed to that the state of the waiting list can be taken out. 9th, method as claimed in claim 7, it is characterised in that it is described the task to be combined is freezed after further comprise：
    If the grade of load of the current system drops to high capacity, middle load or low-load by saturation, middle load or low-load are dropped to by high capacity or drops to low-load by middle load, the task to be combined is changed to that the state of the waiting list can be taken out；Or
    If the urgency level grade of the task to be combined rises to common or urgent by non-emergent, or rise to urgent by common, then by the task to be combined be changed to that the state of the waiting list can be taken out；Or
    When all tasks to be combined are freezed and merge task execution module in idle condition in the waiting list, if the urgency level grade of the urgency level grade task more to be combined than other of the task to be combined is high, the task to be combined is changed to that the state of the waiting list can be taken out.
    10th, a kind of data merging device of key value database, it is characterised in that described device includes：Acquisition module, for obtaining the grade of load of current system and the urgency level grade of task to be combined；Determining module, it is higher for the grade of load of current system that urgency level for the grade of load according to the current system and task to be combined etc. merges tasks carrying strategy, allow to perform urgency level higher grade task to be combined, when the grade of load of current system is saturation, do not allow to perform any task to be combined；Processing module, if determining to perform the task to be combined for the determining module, the task to be combined is sent to merging task execution module and performs task merging.
    11st, device as claimed in claim 10, it is characterised in that the acquisition module includes：First acquisition unit, for the service condition of the resource of current system to be weighted into summation according to weight, load of the value to the current system according to obtained by the weighted sum assigns corresponding grade, the service condition of the resource of the system includes any combination of CPU service condition, internal memory service condition, Internet Use and disk service condition, and the grade of load of the current system includes low-load, middle load, high capacity and saturation successively according to low to high.
    12nd, device as claimed in claim 10, it is characterised in that the acquisition module includes：Second acquisition unit, the factor for the urgency level by the task to be combined is influenceed carries out linear weighted function summation according to weight, urgent journey of the value according to obtained by summing the linear weighted function to the task to be combined Degree assigns corresponding grade, each factor of the urgency level of the influence task to be combined includes any combination of the time of exceeding the time limit of oldest data in file number in the file size in sublist, sublist and sublist, and the urgency level grade of the task to be combined is including common, urgent and necessary.
    13rd, device as claimed in claim 10, it is characterised in that if the influence value of the factor in currently influenceing each factor of the urgency level of the task to be combined has reached or that close to the maximum of the factor acquisition module includes：
    3rd acquiring unit, each factor for the urgency level by the task to be combined is influenceed carries out nonlinear weight summation according to weight, value according to obtained by the weighted sum assigns corresponding grade to the urgency level of the task to be combined, each factor of the urgency level of the influence task to be combined includes any combination of the time of exceeding the time limit of oldest data in file number in the file size in sublist, sublist and sublist, and the urgency level grade of the task to be combined is including non-emergent, common and urgent.
    14th, device as claimed in claim 13, it is characterised in that the 3rd acquiring unit includes：Amplifying unit, the weight ^ for the factor of the urgency level by the task to be combined is influenceed is amplified to f { Ak )wk, described is weight magnification function；
    Sum unit, for using the new weight of the factor as the urgency level for influenceing the task to be combined, the factor of the urgency level of the task to be combined will be influenceed to carry out nonlinear weight summation according to the new weight, value according to obtained by summing the nonlinear weight assigns corresponding grade to the urgency level of the task to be combined, and the urgency level grade of the task to be combined is including common, urgent and necessary.
    15th, the device as described in claim 11 to 14 any one, it is characterised in that the determining module includes：
    First determining unit, if the urgency level grade for matching the task to be combined is urgent, when the grade of load of the current system is low-load, middle load or high capacity, it is determined that allow to perform the task to be combined；
    Second determining unit, if the urgency level grade for matching the task to be combined is non-emergent, when the grade of load of the current system is middle load or the middle load above, it is determined that do not allow to perform the task to be combined；
    3rd determining unit, if the urgency level grade for matching the task to be combined is common, when the grade of load of the current system is more than high capacity or high capacity, it is determined that do not allow to treat described in execution The merging of merging task.
    16th, the device as described in claim 11 to 14 any one, it is characterised in that described device further comprises：
    Task freezes module, for the task to be combined to be freezed, so that the task to be combined is in the freeze-off time of setting can not take out the state of the waiting list.
    17th, device as claimed in claim 16, it is characterised in that described device further comprises：First defrosting module, carries out timing, so as to after the freeze-off time of the setting terminates, the task to be combined is changed to that the state of the waiting list can be taken out for starting timer to the freeze-off time of the setting.
    18th, device as claimed in claim 16, it is characterised in that described device further comprises the second defrosting module, the 3rd defrosting module or the 4th defrosting module；
    The second defrosting module, if the grade of load for the current system drops to high capacity, middle load or low-load by saturation, middle load or low-load are dropped to by high capacity or drops to low-load by middle load, the task to be combined is changed to that the state of the waiting list can be taken out；
    The 3rd defrosting module, if the urgency level grade for the task to be combined rises to common or urgent by non-emergent, or rises to urgent, then by the task to be combined is changed to that the state of the waiting list can be taken out by common；
    The 4th defrosting module, for in the waiting list all tasks to be combined freeze and merge task execution module be in idle condition when, if the urgency level grade of the urgency level grade task more to be combined than other of the task to be combined is high, the task to be combined is changed to that the state of the waiting list can be taken out.
CN201280000718.XA 2012-06-21 2012-06-21 Key-value database data merging method and device Pending CN103649948A (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2012/077314 WO2013189063A1 (en) 2012-06-21 2012-06-21 Key-value database data merging method and device

Publications (1)

Publication Number Publication Date
CN103649948A true CN103649948A (en) 2014-03-19

Family

ID=49768044

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201280000718.XA Pending CN103649948A (en) 2012-06-21 2012-06-21 Key-value database data merging method and device

Country Status (2)

Country Link
CN (1) CN103649948A (en)
WO (1) WO2013189063A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105489216B (en) 2016-01-19 2020-03-03 百度在线网络技术（北京）有限公司 Method and device for optimizing speech synthesis system

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101894163A (en) * 2010-07-23 2010-11-24 中兴通讯股份有限公司 Database operating and scheduling method and device for performance data acquisition system

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101894163A (en) * 2010-07-23 2010-11-24 中兴通讯股份有限公司 Database operating and scheduling method and device for performance data acquisition system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
张晓清等: "《分布式海量数据管理系统Bigtable主服务器设计》", 《计算机工程与设计》 *
王映东等: "《Bigtable系统的负载平衡技术研究》", 《计算机安全》 *

Also Published As

Publication number Publication date
WO2013189063A1 (en) 2013-12-27

Similar Documents

Publication Publication Date Title
US7631034B1 (en) Optimizing node selection when handling client requests for a distributed file system (DFS) based on a dynamically determined performance index
Li et al. A cost-effective mechanism for cloud data reliability management based on proactive replica checking
US9233304B2 (en) Load balancing for game
US8595303B2 (en) Thread data aggregation
CN107515784B (en) Method and equipment for calculating resources in distributed system
JP2012507086A5 (en)
CN104504147A (en) Resource coordination method, device and system for database cluster
CN101923558A (en) Storage network structure and reading and writing method for data based on (d, k) Mohr diagram
KR102450411B1 (en) Method and apparatus for prcocessing transaction based on bllockchain and system tehereof
CA2754086A1 (en) Method and system for transmitting information based on social network
CN108121601B (en) Application resource scheduling device and method based on weight
CN107220348A (en) A kind of method of data capture based on Flume and Alluxio
CN104142953B (en) Metadata dynamic management approach and system
CN104811503A (en) R statistical modeling system
JP2013196389A (en) Information processing apparatus, information processing program and information processing method
CN112163001A (en) High-concurrency query method, intelligent terminal and storage medium
CN103793525A (en) MapReduce model graph node authority value calculation method based on local iteration
Englert et al. Reordering buffer management for non-uniform cost models
CN110209693A (en) High concurrent data query method, apparatus, system, equipment and readable storage medium storing program for executing
CN103841168B (en) Data trnascription update method and meta data server
CN114003345A (en) Cloud-native-based Kubernetes platform health degree determination method and device
CN103649948A (en) Key-value database data merging method and device
CN108572930A (en) Buffer control method and device
CN109040156B (en) Soft load resource processing method and device based on container cloud
CN103455579A (en) Gravitational search model-based copy replacement method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20140319