WO2013189063A1 - Key-value database data merging method and device - Google Patents

Key-value database data merging method and device Download PDF

Info

Publication number
WO2013189063A1
WO2013189063A1 PCT/CN2012/077314 CN2012077314W WO2013189063A1 WO 2013189063 A1 WO2013189063 A1 WO 2013189063A1 CN 2012077314 W CN2012077314 W CN 2012077314W WO 2013189063 A1 WO2013189063 A1 WO 2013189063A1
Authority
WO
WIPO (PCT)
Prior art keywords
merged
task
tasks
urgency
load
Prior art date
Application number
PCT/CN2012/077314
Other languages
French (fr)
Chinese (zh)
Inventor
周帅锋
智伟
赵智峰
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2012/077314 priority Critical patent/WO2013189063A1/en
Priority to CN201280000718.XA priority patent/CN103649948A/en
Publication of WO2013189063A1 publication Critical patent/WO2013189063A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/252Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application

Definitions

  • the present invention relates to the field of data processing, and in particular, to a data merging method and apparatus for a key value database. Background technique
  • the Key-Value database is a database management system based on key-value data storage.
  • Key-value data storage technology stores data in a key-value pair data model that provides persistence mechanisms and data synchronization.
  • data is stored sequentially in terms of keys (Key) and written to disk in chasing mode.
  • BigTable and HBase are the modes. In this mode, files written to the file system are not modified once they are written. Thus, data written at different times will form multiple storage files in the file system.
  • the key-value database will start the merge operation and merge multiple files after the storage file reaches a certain size or for a period of time. At the same time, some data cleaning work is done.
  • sub-tables are the basic unit for performing data merge tasks, and sub-tables are usually managed by multiple sub-table servers.
  • the data table is divided into a number of sub-tables according to the scope of the key. Each sub-table stores and manages all data whose key values fall within its range.
  • the data merging method in a key value database is: using a sub-table as a basic unit of scheduling execution, and when the number of files in the sub-table reaches or exceeds a predefined upper limit, the merging task is triggered, and the sub-task is triggered.
  • the table is placed in the data merge execution queue, from which the data merge thread obtains the child table, and the merge is performed, as shown in FIG.
  • Embodiments of the present invention provide a data merging method and apparatus for a key value database, and control the data merging task to avoid a large amount of data merging tasks while burdening the system.
  • An embodiment of the present invention provides a data merging method for a key value database, where the method includes: acquiring The load level of the current system and the urgency level of the task to be merged;
  • the task to be merged is sent to the merge task execution module to perform task merge.
  • An embodiment of the present invention provides a data merging device for a key value database, where the device includes: an acquiring module, configured to acquire a load level of a current system and an urgency level of a task to be merged;
  • a determining module configured to merge the task execution policy according to the load level of the current system and the urgency of the task to be merged, etc., the higher the load level of the current system, the higher the urgency level is allowed to perform the task to be merged, in the current system When the load level is saturated, the task to be merged is not allowed to be executed; and the processing module is configured to send the to-be-consolidated task to the merge task execution module to perform task combination if the determining module determines to perform the to-be-combined task.
  • the merging is allowed according to the load level of the current system and the urgency level of the task to be merged according to the load level of the current system and the merging task execution strategy, that is, the urgency is allowed according to the system resource load level.
  • the method provided by the embodiment of the present invention can control the data merging task according to the system resource occupation, so that the data merging task is automatically delayed when the system load is high, thereby reducing the impact of the execution task merging on the access service and reducing the system burden.
  • FIG. 1 is a schematic diagram of a data merging method in a key value database provided by the prior art
  • FIG. 2 is a schematic flowchart of a data merging method of a key value database according to an embodiment of the present invention
  • 3 is a schematic structural diagram of a data merging device of a key value database according to an embodiment of the present invention
  • FIG. 4 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 6 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 6b is a data merging device for a key value database according to another embodiment of the present invention
  • FIG. 7 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 7b is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 7 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8 is a structural diagram of a data merging device according to another embodiment of the present invention
  • a schematic diagram of a data merging device of a database FIG. 8b is a key value provided by another embodiment of the present invention
  • FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8
  • FIG. 9 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 9b is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 9 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 10 is a schematic diagram of a data merging database provided by another embodiment of the present invention; Schematic diagram of the data merging device;
  • FIG. 10b is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention.
  • FIG. 10c is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention.
  • Figure 10d is a schematic diagram showing the structure of a data merging device of a key value database according to another embodiment of the present invention.
  • Embodiments of the present invention provide a data merging method and apparatus for a key value database to schedule a data merging task according to system resource occupancy.
  • the technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without departing from the inventive scope are the scope of the present invention.
  • the executor may be a sub-table server in a key value database, and mainly includes step S201, step S202, and step S203:
  • the merge task of the key value database is related to the load of the current system, for example, when the system is busy and/or the system load is relatively high, the unconstrained execution of the key database database merge task will increase the system burden and affect the data access service. Performance.
  • whether or not the merge task of the key-value database is executed is also related to the urgency of the task to be merged. Therefore, in the embodiment of the present invention, the load level of the current system and the urgency level of the task to be merged may be first acquired, where the load level of the current system includes but is not limited to low load, medium load, high load, and saturation.
  • the urgency levels of the tasks to be combined include, but are not limited to, non-emergency, normal, and emergency.
  • the usage of the resources of the current system may be weighted and summed according to the weight, and the load of the current system is given a corresponding level according to the value obtained by the weighted summation.
  • a central processing unit CPU
  • a memory e.g., a RAM
  • a network e.g., a graphics processing unit
  • a disk e.g., a hard disk
  • the weights of the resources may be weighted.
  • the weighted sum is equivalent to the score of the current system load. For example, if the current system's load score is represented by L, then: where ⁇ and ⁇ respectively represent the CPU score weight, the memory score weight, and the network score weight.
  • disk scoring weights, and u cpu u m t ⁇ and respectively indicate CPU usage, memory usage, network usage, and disk usage.
  • the load of the system can be divided into low load, medium load, high load and saturation according to the level from low to high
  • the current system load can be assigned to the corresponding level according to the system load score L from low to high. , for example, if the load on the system When the score L is 100, the load of the system can be scored in the range of 0 ⁇ 25, 25 - 50, 50 ⁇ 75 and 75 ⁇ 100, and the current system load level is sequentially assigned to the low load, medium load, High load and saturation.
  • the factors affecting the urgency of the task to be merged may be linearly weighted and summed according to the weight, and the value to be merged according to the value obtained by the linear weighted summation
  • the urgency of the task corresponds to the level.
  • factors affecting the urgency of the task to be merged include, but are not limited to, the file size in the sub-table, the number of files in the sub-table, and the overdue time of the oldest data in the sub-table. Since there are quantitative differences in different influencing factors, the urgency score of the tasks to be combined can be calculated by linear weighted summation. In this way, for each factor affecting the urgency of the task to be combined, the percentage is calculated based on the maximum value of each factor and weighted and summed, as shown below, if the urgency score of the task to be combined is used
  • Score ⁇ X l , X 2 ,...,X k ) j w k ⁇ X k I MAX Xk ) ⁇ lQQ%
  • represents the first factor affecting the urgency of the task to be merged
  • Indicates the weight of the first factor affecting the urgency of the task to be merged
  • MX indicates the maximum value that the first factor affecting the urgency of the task to be merged can reach or set.
  • the urgency level of the task to be merged can be divided into non-emergency, normal and emergency, etc., it can be scored according to the urgency of the task to be merged ⁇ ( ⁇ ,..., ⁇ )
  • the urgency of the task to be merged is assigned to the corresponding level in turn, for example, if the urgency score of the task to be combined is ⁇ , ⁇ , ..., ⁇ ) is 100, the urgency degree of the task to be combined can be scored. 1 3 ⁇ 4 ⁇ ( 1 , 2 ,..., ⁇ ) In the interval of 0 ⁇ 30, 30 ⁇ 60 and 60 ⁇ 100, the urgency level of the task to be merged is sequentially assigned to non-emergency, normal and emergency.
  • the urgency of the combined task is scored, considering the general state of each factor in the common scenario. However, in some scenarios, the scores of other factors affecting the urgency of the task to be merged may still be If the factors affecting the urgency of the task to be merged are close to or reach the maximum value, the overall impact of the urgency of the execution combination of the tasks to be merged begins to increase non-linearly. Due to the urgency of the formula (2) to treat the combined task Is a linear weighted summation. When the factor affecting the urgency of the task to be merged is close to or reaches the maximum value, if the urgency of the combined task is still scored according to formula (2), the scored result cannot be faithfully reflected. The urgency of the merger to be merged is obviously unreasonable.
  • the embodiment of the invention provides a scoring method for a single item affecting the extreme value amplification weighted summation, so as to achieve a non-linear enlargement of the urgency degree of the combined task when the urgency affecting the task to be combined individually reaches or approaches the maximum value. That is, if the influence value of one of the factors affecting the urgency of the task to be merged has reached or is close to the maximum value of the factor, the factors affecting the urgency of the task to be merged are weighted according to the weight.
  • the nonlinear weighted summation assigns a corresponding level to the urgency of the task to be merged according to the value obtained by the nonlinear weighted summation.
  • the nonlinear weighted summation of each factor affecting the urgency of the task to be merged according to the weight includes: enlarging the weight of the factor affecting the urgency of the task to be merged to a new weight for the urgency of the combined task, a factor that affects the urgency of the task to be merged, a nonlinear weighted summation according to the new weight, and a value pair obtained by the nonlinear weighted summation
  • the urgency level of the task to be merged is the urgency score of the task to be merged ⁇ / ⁇ : ⁇ ,: ⁇ ,...,: ⁇ ;! Expressed as:
  • Equation (4) MAX Xt /(MAX Xt -X k +AX k ) Equation (4)
  • MAX Xt represents the maximum value of the influence factor of the first urgency affecting the task to be combined
  • ⁇ ⁇ is a minimum value of the first factor affecting the urgency of the task to be merged, and is used to adjust equation (4) such that the denominator of equation (4) is not zero.
  • the load level of the current system obtained by the above-mentioned score calculated according to the formula (1), and the urgency level of the task to be combined obtained by the score calculated according to the formula (2) or the formula (3) are only For example, it is not limited to the formula (1), (2) or (3) above. Style.
  • the principle of determining a set of tasks for performing task consolidation includes: The higher the load level of the current system, the higher the urgency level is allowed to perform the task to be merged, that is, the task to be merged is allowed to be higher.
  • the merge operation is performed. When the load level of the current system is saturated, no tasks to be merged are allowed to be executed, that is, the merge of any tasks to be merged is not allowed.
  • the execution strategy of the merged task can also be interpreted as: When the urgency level of the task to be merged is determined, the higher the load level of the current system, the lower the possibility that the task to be merged is allowed to perform the merge, and vice versa, the current system The lower the load level, the higher the probability that the task to be merged is allowed to perform the merge; or the higher the urgency level of the task to be merged when the load level of the current system is determined (ie, the degree to which the task to be merged needs to perform the merge) The more urgent, the higher the probability that the task to be merged is allowed to perform the merge, and the lower the level of urgency of the task to be merged (that is, the more relaxed the degree of the merge to be merged), the task to be merged is not allowed.
  • the urgency level of the tasks to be merged is merged; when the urgency level of the task to be merged is the highest, for example, when the urgency level of the task to be merged is urgent, unless the load level of the current system is the highest, for example, the load of the current system The level is saturated, otherwise, the merging is allowed for the task to be merged with the urgency level being urgent.
  • the current system's load levels include, but are not limited to, low load, medium load, high load, and saturation
  • the urgency levels of the tasks to be combined include, but are not limited to, non-emergency, normal, and emergency.
  • the merge task execution policy provided by the embodiment of the present invention may be as shown in Table 1 below:
  • the "load level” is the cartridge of the "current system load level” in the foregoing embodiment, including “low load”, “medium load”, “high load” and “saturated”, “urgency level”.
  • " is the nickname of the urgency level of the task to be merged in the foregoing embodiment, including “non-emergency", “normal”, and “emergency”.
  • the item at the intersection of "load level” and “urgency level” indicates whether to perform the combination of the tasks to be merged. For example, when the "load level” is “low load” and the “urgency level” is “non-emergency", Perform a merge of the tasks to be merged.
  • the general principle of the combined task execution strategy reflected in Table 1 above is that for a task to be merged with a urgency level of non-emergency, when the load level of the system is above the medium load or the medium load, the merged service is not allowed to be merged.
  • the task to be merged is returned to the queue and waits; for the task to be merged with the urgency level is normal, when the load level of the system is high load or high load, the merged service is not allowed to be merged, but
  • the to-be-consolidated task returns to the queue to wait; for the to-be-consolidated task whose urgency level is urgent, as long as the load level of the system is not saturated, that is, when the current system load level is low load, medium load or high load, the to-be-committed is allowed.
  • Business execution merges.
  • matching the merge task execution policy to determine whether to allow the task to be merged includes: if the urgency level matching the task to be merged is urgent, When the load level of the current system is low load, medium load or high load, it is determined that the task to be merged is allowed to be executed, that is, it is determined that the merge is allowed to be performed on the task to be merged; if the task is to be merged to the task to be merged If the level of the current system is non-emergency, when the load level of the current system is above the medium load or the medium load, it is determined that the task to be merged is not allowed to be executed, that is, it is determined that the merged task is not allowed to be merged; The urgency level assigned to the task to be merged is normal. When the load level of the current system is high load or high load, it is determined that the task to be merged is not allowed to be executed, that is, the waiting for the waiting is not allowed.
  • the merge task execution policy of the example in Table 1 For the merge task execution policy of the example in Table 1, if the load level of the current system acquired in step S201 is "low load”, the merge task execution policy of the example of Table 1 is matched, regardless of the urgency of the task to be merged acquired in step S201. Whether the level is "non-emergency", "normal” or “emergency”, it is determined that the combination of the tasks to be merged can be performed; if the load level of the current system acquired in step S201 is "medium load", it matches the example of Table 1.
  • the merging task execution policy when the urgency level of the task to be merged obtained in step S201 is "normal” or “emergency”, it is also determined that the merging of the tasks to be merged can be performed; if the load level of the current system acquired in step S201 is "High load", then matching to the merge task execution policy of the example in Table 1, only when the urgency level of the task to be merged acquired in step S201 is "emergency", it is determined that the merge of the task to be merged can be performed; When the load level of the current system acquired by S201 is "saturated", it matches the merge task execution of the example in Table 1.
  • step S201 regardless of whether the urgency level of the task to be merged obtained in step S201 is "non-emergency", "normal” or “emergency", it is determined that the merging of the tasks to be merged cannot be performed; if the load level of the current system acquired in step S201 The "high load” is matched to the merge task execution policy of the example in Table 1. When the urgency level of the task to be merged acquired in step S201 is "non-emergency" or "normal”, it is also determined that the to-be-committed may not be executed.
  • step S201 If the load level of the current system acquired in step S201 is "medium load”, the matching task execution policy of the example in Table 1 is matched, and when the urgency level of the task to be merged acquired in step S201 is "non-emergency" It is also determined that the merging of the tasks to be merged cannot be performed.
  • the merge task execution module is a functional unit that performs task merge, and one or more such functional units may exist in the child table server.
  • the merged task execution module is idle, the task can be performed, and the task is actively requested from the scheduler, or the active report is idle, and the scheduler is assigned to the task. Therefore, according to step S201 and step S202, if it is determined that the task to be merged can be executed, that is, it is determined that the merge can be performed on the task to be merged, The merged task is sent to the merge task execution module, and the merge task execution module performs task merge.
  • the to-be-combined task is returned to the waiting queue, and waits for an appropriate timing to be taken out from the waiting queue to perform task combining.
  • the sub-table server is not suitable for performing task consolidation for a relatively long period of time, for example, if the load of the current system is in a long period of time.
  • the merging of tasks If, during the period of the above example, the tasks in the waiting queue cannot be performed according to the merge task execution policy before the queue is fetched frequently from the waiting queue, it is obviously a waste of resources.
  • step S201 and step S202 After the merge task execution policy according to the example in Table 1 cannot perform the task merge and returns to the waiting queue in the waiting queue, the task to be merged may be frozen, so that the task to be merged is within the set freeze time. It is in a state in which the waiting queue cannot be taken out.
  • the size of the freeze time setting may be set by the user according to the load level of the current system and/or the expected duration of the urgency level of the task to be merged, which is not particularly limited in the present invention.
  • a timer may be started to time the set freeze time so that after the set freeze time is over,
  • the merge task execution policy according to the table 1 cannot perform the task merge and the task to be merged in the wait queue is changed to the state in which the wait queue can be taken out, thereby reevaluating whether the merge task can perform the merge.
  • the freeze time is set based on the length of the expected duration of the current system's load level, it is also impossible to exclude that the freeze time setting exceeds the expected duration of the current system load level. For example, if the load level of the current system or/and the urgency level of the task to be merged changes, thereby performing the merge task execution strategy as exemplified in Table 1, the previous example according to Table 1 And the task execution strategy can not perform the merged to-be-merge task, and the merge can be re-executed. At this time, if the mechanical merge is waited for the set freeze time to end, the task merge is performed, which is also a waste of resources for the system.
  • a change occurs in the load level of the current system, that is, if the load level of the current system is decreased from “saturated” to "high load”, “medium load” or “low load”, Then, in step S201 and step S202, the merge task execution policy according to the table 1 may not perform task merge, and the task to be merged in the wait queue is changed to a state in which the wait queue can be taken out; or, if the load of the current system is If the level is lowered from "high load” to "medium load” or "low load”, the merge task execution strategy according to Table 1 in step S201 and step S202 can not perform task merge and return to the task to be merged in the wait queue.
  • the change of the urgency level of the task to be merged that is, if the merged task execution policy according to the table 1 in step S201 and step S202 cannot perform the task merging and returns to the waiting queue If the urgency level of the merging task is raised from “non-emergency" to "normal” or “emergency", the task to be merged is changed to a state in which the waiting queue can be taken out; or, if in step S201 and step S202, according to Table 1
  • the example merge task execution policy cannot perform task merge and the urgency level of the task to be merged in the wait queue is raised from "normal” to "emergency", and the task to be merged is changed to a state in which the wait queue can be taken out.
  • the merge task execution strategy according to Table 1 is performed in steps S201 and S202 If the urgency level of the task to be merged in the waiting queue cannot be performed and the urgency level of the task to be merged is higher than that of the other tasks to be merged, the merged task execution policy according to the example in Table 1 in step S201 and step S202 cannot be executed.
  • the tasks to be merged and returned to the waiting queue are changed to a state in which the waiting queue can be taken out.
  • the method provided by the embodiment of the present invention can be based on the prior art without considering the load or the state of the system (for example, the system is relatively busy and/or the system load is relatively high) and the merge of the tasks to be merged is still performed without limitation.
  • the system resource occupation situation is used to schedule the data merge task, so that the data merge task is automatically delayed when the system load is high, and is automatically executed when the system is under low load, thereby reducing the impact of the execution task merge on the access service, and allowing the task to be merged.
  • the merger is performed automatically and in a timely manner at the right time.
  • FIG. 3 it is a structural diagram of a data merging device of a key value database according to an embodiment of the present invention. For the convenience of description, only parts related to the embodiment of the present invention are shown.
  • the data combining apparatus of the key value database illustrated in Fig. 3 includes an obtaining module 301, a determining module 302, and a processing module 303, wherein:
  • the obtaining module 301 is configured to acquire a load level of the current system and an urgency level of the task to be merged.
  • the merge task of the key value database is related to the load of the current system, for example, when the system is busy and/or the system load is relatively high, the unrestricted execution of the key database merge task will increase the system burden and affect the data access service. Performance.
  • whether or not the merge task of the key-value database is executed is also related to the urgency of the task to be merged. Therefore, in this embodiment, the level of the current system and the level of urgency of the task to be merged may be first acquired by the level obtaining module 301, where the load level of the current system includes but is not limited to low load, medium load, high load, and There are several types of saturation, such as non-emergency, normal and emergency.
  • the determining module 302 is configured to use, according to the load level of the current system and the emergency process of the task to be merged, in the embodiment, the combined task execution policy is pre-defined according to the load level of the system and the urgency level of the task to be merged.
  • the principle of determining whether to perform task consolidation includes the following examples: the higher the load level of the current system, the higher the urgency task to be merged, that is, the higher the urgency level is to be performed. Merging, when the current system's load level is saturated, it is not allowed to perform any tasks to be merged, that is, it is not allowed to perform the merge of any tasks to be merged.
  • the execution policy of the merged task may also be interpreted as: when the urgency level of the task to be merged is determined, the higher the load level of the current system, the task to be merged is allowed to execute.
  • the lower the probability of the sum the lower the load level of the current system, the higher the probability that the task to be merged is allowed to perform the merge; or the urgency level of the task to be merged when the load level of the current system is determined
  • the higher the degree ie, the more urgent the task to be merged needs to perform the merge
  • the lower the level of urgency of the task to be merged ie, the task to be merged needs to perform the merge
  • the more relaxed the degree the higher the probability that the task to be merged is not allowed to perform the merge; when the load level of the current system is the lowest, for example, when the load level of the current system is low load, the urgency level can be merged.
  • Task execution merge When the current system's load level is the highest, for example, when the current system's load level is saturated, it is not allowed to perform merging for any urgency level to be merged task; when the urgency level of the task to be merged is the highest, for example , the urgency level of the task to be merged is urgent, unless the current system
  • the load level is the highest, for example, the load level of the current system is saturated, otherwise, the merge is allowed to be performed on the task to be merged whose emergency level is urgent.
  • the current system's load levels include, but are not limited to, low load, medium load, high load, and saturation
  • the urgency levels of the tasks to be combined include, but are not limited to, non-emergency, normal, and emergency.
  • the merge task execution policy provided in this embodiment may be as shown in Table 1 above.
  • the processing module 303 is configured to: if the determining module 302 determines to perform the to-be-committed task, send the to-be-combined task to the merged task execution module to perform task combining.
  • the merge task execution module is a functional unit that performs task merge, and one or more such functional units may exist in the child table server.
  • the merged task execution module is idle, the task can be performed, and the task is actively requested from the scheduler, or the active report is idle, and the scheduler is assigned to the task. Therefore, once the determining module 302 determines that the to-be-combined task can be performed, that is, it is determined that the merge can be performed on the to-be-combined task, the processing module 303 sends the to-be-combined task to the merged task execution module, which is executed by the merged task execution module. Task consolidation.
  • the processing module 303 may further return the to-be-combined task to the waiting queue, wait for a suitable timing, and wait for another time. Take out the queue to perform task consolidation.
  • each functional module is only an example, and the actual application may be configured according to requirements, for example, corresponding hardware requirements.
  • the convenience of the implementation of the software, and the above-mentioned function allocation is completed by different functional modules, that is, the internal structure of the data combining device of the key value database is divided into different functional modules to complete all or part of the functions described above.
  • the corresponding functional modules in this embodiment may be implemented by corresponding hardware, or may be executed by corresponding hardware.
  • the foregoing level obtaining module may have the foregoing acquisition current
  • the hardware of the load level of the system and the urgency level of the task to be merged such as a level acquirer, may also be a general processor or other hardware device capable of executing a corresponding computer program to perform the aforementioned functions;
  • An emergency program, such as a determiner, that performs the foregoing load level according to the current system and a task to be merged, such as a determiner, may also be a general processor or other hardware device capable of executing a corresponding computer program to perform the aforementioned functions (each provided in this specification)
  • the above described principles can be applied to the embodiments).
  • the acquisition module 301 of the example of FIG. 3 may include a first acquisition unit 401, such as the data integration device of the key value database provided by another embodiment of the present invention as shown in FIG.
  • the first obtaining unit 401 is configured to perform a weighted summation of the usage of the resources included in the current system according to the weight, and assign a corresponding level to the load of the current system according to the value obtained by the weighted summation, where the system is Resource usage includes any combination of CPU usage, memory usage, network usage, and disk usage, including current load, medium load, high load, and saturation.
  • a central processing unit (CPU), a memory, a network, a disk, and the like all belong to resources included in the current system.
  • the first obtaining unit 401 may perform weighted summation according to the weight of each resource after acquiring the usage of the resources, and the weighted summation value is equivalent to scoring the current system load, for example, if the current system load score When L is used, the expression of L is as shown in the above formula (1). If the load of the system can be divided into low load, medium load, high load and saturation according to the level from low to high, the first obtaining unit 401 can load the current system according to the load score L of the system from low to high. The corresponding level is assigned in turn.
  • the first acquisition unit is when the load score of the system is in the range of 0 to 25, 25 to 50, 50 to 75, and 75 to 100.
  • 401 correspondingly sequentially assigns the load level of the current system to low load, medium load, high load, and saturation.
  • the acquiring module 301 may further include other functional modules or units for acquiring the urgency level of the task to be merged, and the obtaining manner thereof is in the drawing. 4
  • the device of the example is not limited.
  • the level obtaining module 301 of the example of FIG. 3 may also include a second obtaining unit 501, such as the data combining device of the key value database provided by another embodiment of the present invention as shown in FIG.
  • the second obtaining unit 501 is configured to linearly weight the factors affecting the urgency of the task to be merged according to the weight, and assign a corresponding level to the urgency of the task to be merged according to the value obtained by the linear weighted summation.
  • the factors affecting the urgency of the task to be merged include any combination of the file size in the sub-table, the number of files in the sub-table, and the expiration time of the oldest data in the sub-table, where the task to be merged Urgency levels include general, urgent, and mandatory.
  • factors affecting the urgency of the task to be merged include, but are not limited to, the file size in the sub-table, the number of files in the sub-table, and the overdue time of the oldest data in the sub-table. and many more. Since there are quantitative differences in different influencing factors, the second obtaining unit 501 can calculate the urgency score of the task to be merged by means of linear weighted summation. In this way, for each factor affecting the urgency of the task to be combined, the percentage is calculated according to the maximum value of each factor and weighted and summed. If the urgency score Srar of the task to be combined is expressed by the above formula (2) The urgency level of the task to be merged may be divided into normal, urgent, and mandatory, and the second obtaining unit 501 may score according to the urgency of the task to be merged.
  • the second obtaining unit 501 sequentially assigns the urgency level of the task to be combined to ordinary, urgent, and necessary .
  • the obtaining module 301 may further include other functional modules or units for acquiring the load level of the current system, and the obtaining manner thereof is illustrated in FIG. 5.
  • the device is not limited.
  • the urgency of the combined task is scored, considering the general state of each factor in the common scenario, however, in some scenarios, despite other emergency affecting the task to be merged
  • the score of the degree factor may still be relatively low, but when the factor affecting the urgency of the task to be merged is close to or reaches the maximum value, the overall impact of the urgency of the execution combination of the task to be merged begins to be nonlinear.
  • the ground is enlarged. Since the urgency of the merging task is evaluated by the formula (2), the linear weighted summation is used.
  • the second obtaining unit 501 still follows the formula (2).
  • the acquisition module 301 exemplified in FIG. 3 may also include a third acquisition unit 601, as shown in FIG. 6a, which is a key value database provided by another embodiment of the present invention. Data merge device.
  • a third obtaining unit 601 configured to perform non-linear weighted summation of each factor affecting the urgency of the task to be merged according to a weight, and assign a urgency to the task to be merged according to the value obtained by the weighted summation
  • the level of the urgency affecting the task to be merged includes any combination of the file size in the child table, the number of files in the child table, and the time of the oldest data in the child table, the task to be merged Urgency levels include non-emergency, general and emergency.
  • the obtaining module 301 may further include other functional modules or units for acquiring the load level of the current system, and the obtaining manner thereof is illustrated in FIG. 6a.
  • the device is not limited.
  • the third obtaining unit 601 illustrated in Fig. 6a may further include an amplifying unit 6011 and a summing unit
  • a data merging device for a key value database according to another embodiment of the present invention, wherein:
  • the amplifying unit 6011 is configured to expand the weight of the factor affecting the urgency of the task to be merged to the weight amplifying function.
  • a summation unit 6012 configured to weight the factors affecting the urgency of the task to be merged according to the new weight according to the new weight as a factor affecting the urgency of the task to be merged, according to the new weight a value corresponding to the urgency of the task to be merged, that is, a summation unit 6012 assigns a urgency to the task to be merged according to formula (3)
  • the level of responsibility including general, urgent, and must, and so on.
  • the determining module 302 illustrated in FIG. 4 to FIG. 6b may include a first determining unit 701, a second determining unit 702, and a third determining unit 703, as shown in FIG. 7a to FIG. 7d, which are provided by another embodiment of the present invention.
  • a data merging device for a key value database where:
  • the first determining unit 701 is configured to: if the urgency level of the task to be merged is urgent, when the load level of the current system is low load, medium load, or high load, determine to allow execution of the to-be-committed The merging of tasks.
  • the acquisition module 301 matches the merge task execution policy of the example in Table 1.
  • the load level of the current system acquired by the acquisition module 301 is "low load”.
  • the first determining unit 701 determines that the task to be merged is allowed to be executed, that is, determines that the merging is allowed to be performed on the task to be merged, or the load level of the current system acquired by the obtaining module 301 is “medium load”, and the first determining unit 701 determines
  • the task to be merged is allowed to be executed, that is, it is determined that the merging is allowed to be performed on the to-be-combined task, or the load level of the current system acquired by the obtaining module 301 is “high load”, and the first determining unit 701 determines that the to-be-committed is allowed to be executed.
  • the task ie determining to allow the merging of the tasks to be merged.
  • the second determining unit 702 is configured to: if the urgency level of the task to be merged is non-emergency, when the load level of the current system is greater than the medium load or the medium load, determine that the to-be-committed is not allowed to be performed. The task, that is, determines that the merging of the tasks to be merged is not allowed.
  • the acquisition module 301 matches the merge task execution policy of the example in Table 1.
  • the load level of the current system acquired by the acquisition module 301 is "medium load”.
  • the second determining unit 702 determines that the to-be-combined task is not allowed to be executed, that is, it is determined that the merging is not allowed to be performed on the to-be-joined task, or the load level of the current system acquired by the obtaining module 301 is “high load”, and the second determination is performed.
  • the unit 702 determines that the task to be merged is not allowed to be executed, that is, it is determined that the merge is not allowed to be performed on the task to be merged, or the load level of the current system acquired by the obtaining module 301 is "saturated", and the second determining unit 702 determines that the task is not allowed.
  • the tasks to be merged are executed, that is, it is determined that the merge is not allowed to be performed on the tasks to be merged.
  • the third determining unit 703 is configured to: if the urgency level of the task to be merged is normal, when the load level of the current system is higher than a high load or a high load, determine that the performing is not allowed To be merged.
  • the merge task execution policy of the example of Table 1 is matched, and the load level of the current system acquired by the obtaining module 301 is "
  • the third determining unit 703 determines that the to-be-combined task is not allowed to be executed, that is, it is determined that the merging of the to-be-combined task is not allowed.
  • the data merging apparatus of the key value database illustrated in Figures 4 through 6b may include a task freeze module 801, such as the data merging apparatus of the key value database provided by another embodiment of the present invention, as shown in Figures 8a to 8d.
  • the task freeze module 801 is configured to freeze the to-be-consolidated task, so that the to-be-combined task is in a state in which the waiting queue cannot be taken out within a set freeze time.
  • the sub-table server should not perform task consolidation for a relatively long period of time, for example, if the current system load is in a long period of time
  • the task freezing module 801 may freeze the to-be-consolidated task to make the waiting The merge task is in a state in which the wait queue cannot be fetched within the set freeze time.
  • the size of the freeze time setting may be set by the user according to the load level of the current system and/or the expected duration of the urgency level of the task to be merged, which is not specifically limited by the present invention.
  • the data merging device of the key value database illustrated in Figures 8a to 8d may include a first defrosting module 901, such as the data merging device of the key value database provided by another embodiment of the present invention as shown in Figures 9a to 9d.
  • the first defrosting module 901 is configured to start a timer to time the set freezing time, so that after the set freezing time ends, the to-be-combined task is changed to a state in which the waiting queue can be taken out. . Since the freeze time is set based on the length of the expected duration of the current system's load level, it is also impossible to exclude that the freeze time setting exceeds the expected duration of the current system load level.
  • the data merging device of the key value database illustrated in FIGS. 8a to 8d may also include a second defrosting module 1001, a third defrosting module 1002, or a fourth defrosting module 1003, as shown in FIG. 10a to FIG. 10d.
  • a data merging device for a key value database provided by another embodiment, wherein:
  • the second defrosting module 1001 is configured to: if the load level of the current system decreases from saturation to high load, medium load or low load, from high load to medium load or low load, or from medium load to low load, The task to be merged is changed to a state in which the waiting queue can be taken out.
  • the third defrosting module 1002 is configured to change the to-be-combined task to be able to take out the waiting queue if the urgency level of the task to be merged rises from non-emergency to normal or urgent, or from ordinary to urgent. status.
  • the fourth defrosting module 1003 is configured to: when all the tasks to be merged are frozen in the waiting queue and the merging task execution module is in an idle state, if the urgency level of the task to be merged is higher than the urgency level of other tasks to be merged And changing the to-be-combined task to a state in which the waiting queue can be taken out.
  • the to-be-combined task is sent to the merged task execution module to perform a merge task.
  • the program may be stored in a computer readable storage medium, and the storage medium may include: Read Only Memory (ROM), Random Access Memory (RAM), disk or optical disk.
  • ROM Read Only Memory
  • RAM Random Access Memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present invention provides a key-value database data merging method and device, to control a data merging task, so as to prevent a heavier burden on the system incurred by concurrence of a large number of data merging tasks. The method comprises: obtaining a load level of a current system and an urgency level of a to-be-merged task; according to the load level of the current system and the urgency level of the to-be-merged task, matching a task-merging execution policy to determine whether the to-be-merged task is permitted to be executed; if it is determined that the to-be-merged task is permitted to be executed, sending the to-be-merged task to a task-merging execution module to execute task merging. The method provided by the present invention can control the data merging task according to the occupation of the system resources, so as to make the data merging task be automatically delayed when the load of the system is high, thereby reducing the impact of execution of task merging on service accessing, and reducing the burden on the system.

Description

键值数据库的数据合并方法和装置 技术领域  Data merging method and device for key value database
本发明涉及数据处理领域, 尤其涉及键值数据库的数据合并方法和装置。 背景技术  The present invention relates to the field of data processing, and in particular, to a data merging method and apparatus for a key value database. Background technique
键值 ( Key- Value )数据库是以键 -值数据存储为基础的数据库管理系统。 键值数据存储技术以键值对的数据模型存储数据,能够提供持久化机制和数据 同步等功能。 在一些键值数据库中, 数据是按照键(Key ) 以顺序存储并以追 加模式写入磁盘, 例如, BigTable和 HBase就是这种模式。 在这种模式下, 写 入文件系统的文件, 一经写入后, 不会修改。 如此, 不同时期写入的数据会在 文件系统中形成多个存储文件。 为了便于访问、保持数据的有序和及时清理过 期数据, 以及提高数据访问的速度等目的,键值数据库会在存储文件达到一定 规模或者持续一段时间之后, 启动合并操作, 将多个文件进行合并, 同时进行 一些数据清理工作。  The Key-Value database is a database management system based on key-value data storage. Key-value data storage technology stores data in a key-value pair data model that provides persistence mechanisms and data synchronization. In some key-value databases, data is stored sequentially in terms of keys (Key) and written to disk in chasing mode. For example, BigTable and HBase are the modes. In this mode, files written to the file system are not modified once they are written. Thus, data written at different times will form multiple storage files in the file system. In order to facilitate access, maintain orderly data and timely clean up expired data, and improve the speed of data access, the key-value database will start the merge operation and merge multiple files after the storage file reaches a certain size or for a period of time. At the same time, some data cleaning work is done.
在一些大容量分布式的键值数据库中,子表是执行数据合并任务的基本单 位, 通常由多个子表服务器来管理子表。 为了便于管理, 根据键(Key ) 的范 围,将数据表划分为很多子表,每个子表存储并管理键值落在其范围之内的所 有数据。  In some large-capacity distributed key-value databases, sub-tables are the basic unit for performing data merge tasks, and sub-tables are usually managed by multiple sub-table servers. For ease of management, the data table is divided into a number of sub-tables according to the scope of the key. Each sub-table stores and manages all data whose key values fall within its range.
现有技术提供的一种键值数据库中数据合并方法是:以子表为调度执行的 基本单位, 当子表中的文件个数达到或者超过预先定义的上限之后,触发合并 任务, 将该子表放入数据合并执行队列, 由数据合并线程从中获取子表, 执行 合并, 如附图 1所示。  The data merging method in a key value database provided by the prior art is: using a sub-table as a basic unit of scheduling execution, and when the number of files in the sub-table reaches or exceeds a predefined upper limit, the merging task is triggered, and the sub-task is triggered. The table is placed in the data merge execution queue, from which the data merge thread obtains the child table, and the merge is performed, as shown in FIG.
由于执行数据合并任务的一系列过程通常会耗费大量的系统资源,所有的 合并任务无限制地执行显然会加重系统负担, 影响数据访问业务的性能。 发明内容  Since a series of processes for performing data merge tasks usually consume a large amount of system resources, the unrestricted execution of all the merge tasks obviously increases the system burden and affects the performance of the data access service. Summary of the invention
本发明实施例提供键值数据库的数据合并方法和装置,对数据合并任务进 行控制, 以避免大量数据合并任务同时进行加重系统负担。  Embodiments of the present invention provide a data merging method and apparatus for a key value database, and control the data merging task to avoid a large amount of data merging tasks while burdening the system.
本发明实施例提供一种键值数据库的数据合并方法, 所述方法包括: 获取 当前系统的负载等级和待合并任务的紧急程度等级; An embodiment of the present invention provides a data merging method for a key value database, where the method includes: acquiring The load level of the current system and the urgency level of the task to be merged;
根据所述当前系统的负载等级和待合并任务的紧急程度等级,匹配合并任 略为当前系统的负载等级越高, 允许执行紧急程度等级越高的待合并任务, 在 当前系统的负载等级为饱和时, 不允许执行任何待合并任务;  According to the current system load level and the urgency level of the task to be merged, the higher the load level of the current system is, the higher the urgency level is allowed to perform the task to be merged, when the current system load level is saturated. , does not allow any tasks to be merged;
若确定允许执行所述待合并任务,则将所述待合并任务发送至合并任务执 行模块执行任务合并。  If it is determined that the task to be merged is allowed to be executed, the task to be merged is sent to the merge task execution module to perform task merge.
本发明实施例提供一种键值数据库的数据合并装置, 所述装置包括: 获取 模块, 用于获取当前系统的负载等级和待合并任务的紧急程度等级;  An embodiment of the present invention provides a data merging device for a key value database, where the device includes: an acquiring module, configured to acquire a load level of a current system and an urgency level of a task to be merged;
确定模块,用于根据所述当前系统的负载等级和待合并任务的紧急程度等 合并任务执行策略为当前系统的负载等级越高,允许执行紧急程度等级越高的 待合并任务, 在当前系统的负载等级为饱和时, 不允许执行任何待合并任务; 处理模块, 用于若所述确定模块确定执行所述待合并任务, 则将所述待合 并任务发送至合并任务执行模块执行任务合并。  a determining module, configured to merge the task execution policy according to the load level of the current system and the urgency of the task to be merged, etc., the higher the load level of the current system, the higher the urgency level is allowed to perform the task to be merged, in the current system When the load level is saturated, the task to be merged is not allowed to be executed; and the processing module is configured to send the to-be-consolidated task to the merge task execution module to perform task combination if the determining module determines to perform the to-be-combined task.
从上述本发明实施例可知,由于待合并任务是否执行合并是根据当前系统 的负载等级和待合并任务的紧急程度等级, 匹配合并任务执行策略来确定, 即 根据系统资源负载等级, 允许执行紧急程度高的待合并任务。 因此, 本发明实 施例提供的方法能够根据系统资源占用情况来控制数据合并任务,让数据合并 任务在系统负载高时自动延迟, 从而达到减小执行任务合并对访问业务影响, 减轻系统负担。  According to the embodiment of the present invention, the merging is allowed according to the load level of the current system and the urgency level of the task to be merged according to the load level of the current system and the merging task execution strategy, that is, the urgency is allowed according to the system resource load level. High task to be merged. Therefore, the method provided by the embodiment of the present invention can control the data merging task according to the system resource occupation, so that the data merging task is automatically delayed when the system load is high, thereby reducing the impact of the execution task merging on the access service and reducing the system burden.
附图说明 DRAWINGS
为了更清楚地说明本发明实施例的技术方案,下面将对现有技术或实施例 描述中所需要使用的附图作筒单地介绍,显而易见地, 下面描述中的附图仅仅 是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动 性的前提下, 还可以如这些附图获得其他的附图。  In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the prior art or the embodiments will be briefly described below. Obviously, the drawings in the following description are only the present invention. For some embodiments, other drawings may be obtained as those of ordinary skill in the art without any inventive labor.
图 1是现有技术提供的一种键值数据库中数据合并方法示意图;  1 is a schematic diagram of a data merging method in a key value database provided by the prior art;
图 2是本发明实施例提供的键值数据库的数据合并方法流程示意图; 图 3是本发明实施例提供的键值数据库的数据合并装置结构示意图; 图 4是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 5是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 6a是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 6b是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 7a是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 7b是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 7c是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 7d是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 8a是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 8b是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 8c是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 8d是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 9a是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 9b是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 9c是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 9d是本发明另一实施例提供的键值数据库的数据合并装置结构示意图; 图 10a是本发明另一实施例提供的键值数据库的数据合并装置结构示意 图; 2 is a schematic flowchart of a data merging method of a key value database according to an embodiment of the present invention; 3 is a schematic structural diagram of a data merging device of a key value database according to an embodiment of the present invention; FIG. 4 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 6 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 6b is a data merging device for a key value database according to another embodiment of the present invention; FIG. 7 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 7b is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 7 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 8 is a structural diagram of a data merging device according to another embodiment of the present invention; A schematic diagram of a data merging device of a database; FIG. 8b is a key value provided by another embodiment of the present invention; FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 9 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 9b is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 9 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG. 10 is a schematic diagram of a data merging database provided by another embodiment of the present invention; Schematic diagram of the data merging device;
图 10b是本发明另一实施例提供的键值数据库的数据合并装置结构示意 图;  FIG. 10b is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG.
图 10c是本发明另一实施例提供的键值数据库的数据合并装置结构示意 图;  FIG. 10c is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention; FIG.
图 10d是本发明另一实施例提供的键值数据库的数据合并装置结构示意 图。  Figure 10d is a schematic diagram showing the structure of a data merging device of a key value database according to another embodiment of the present invention.
具体实施方式 detailed description
本发明实施例提供键值数据库的数据合并方法和装置,以根据系统资源占 用情况来调度数据合并任务。 下面将结合本发明实施例中的附图,对本发明实施例中的技术方案进行清 楚、 完整地描述, 显然, 所描述的实施例仅仅是本发明一部分实施例, 而不是 全部的实施例。基于本发明中的实施例, 本领域普通技术人员在没有做出创造 性劳动前提下所获得的所有其他实施例, 都属于本发明保护的范围。 Embodiments of the present invention provide a data merging method and apparatus for a key value database to schedule a data merging task according to system resource occupancy. The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without departing from the inventive scope are the scope of the present invention.
请参阅附图 2, 是本发明实施例提供的键值数据库的数据合并方法流程示 意图, 其执行主体可以是键值数据库中的子表服务器, 主要包括步骤 S201、 步 骤 S202和步骤 S203:  Referring to FIG. 2, it is a flowchart of a data merging method of a key value database according to an embodiment of the present invention. The executor may be a sub-table server in a key value database, and mainly includes step S201, step S202, and step S203:
S201 , 获取当前系统的负载等级和待合并任务的紧急程度等级。  S201. Obtain a load level of the current system and an urgency level of the task to be merged.
由于键值数据库的合并任务是否执行与当前系统的负载有关, 例如,在系 统比较繁忙和 /或系统负载比较高时, 无限制地执行键值数据库的合并任务会 加重系统负担, 影响数据访问业务的性能。 除此之外, 键值数据库的合并任务 是否执行还与待合并任务的紧急程度有关。 因此, 在本发明实施例中, 可以首 先获取当前系统的负载等级和待合并任务的紧急程度等级, 其中, 当前系统的 负载等级包括但不限于低负载、 中负载、 高负载和饱和等几种, 而待合并任务 的紧急程度等级包括但不限于非紧急、 普通和紧急等几种。  Whether the merge task of the key value database is related to the load of the current system, for example, when the system is busy and/or the system load is relatively high, the unconstrained execution of the key database database merge task will increase the system burden and affect the data access service. Performance. In addition to this, whether or not the merge task of the key-value database is executed is also related to the urgency of the task to be merged. Therefore, in the embodiment of the present invention, the load level of the current system and the urgency level of the task to be merged may be first acquired, where the load level of the current system includes but is not limited to low load, medium load, high load, and saturation. The urgency levels of the tasks to be combined include, but are not limited to, non-emergency, normal, and emergency.
作为获取当前系统的负载等级的一个实施例,可以将当前系统的资源的使 用情况按照权重进行加权求和,根据所述加权求和所得的值对所述当前系统的 负载赋予相应的等级。在本发明实施例中,中央处理器( Central Processing Unit, CPU ), 内存、 网络和磁盘等都属于当前系统包含的资源, 可以在获取这些资 源的使用情况后,按照各个资源的权重进行加权求和,加权求和所得的值相当 于是对当前系统的负载进行的评分,例如,若当前系统的负载评分使用 L表示, 则: 其中, 和^分別表示 CPU评分权重、 内存评分权重、 网络评 分权重和磁盘评分权重, 而 ucpu um t ^和 分別表示 CPU的使用情况、 内存使用情况、 网络使用情况和磁盘使用情况。若系统的负载按照等级由低到 高可依次分为低负载、 中负载、 高负载和饱和等几种, 则可以按照系统的负载 评分 L从低到高将当前系统的负载依次赋予相应的等级, 例如, 若系统的负载 评分 L满分为 100, 则可以将系统的负载评分 在0 ~ 25、 25 -50, 50 ~ 75和 75 ~ 100四个区间时, 相应地依次将当前系统的负载等级赋予低负载、 中负载、 高 负载和饱和。 As an embodiment of acquiring the load level of the current system, the usage of the resources of the current system may be weighted and summed according to the weight, and the load of the current system is given a corresponding level according to the value obtained by the weighted summation. In the embodiment of the present invention, a central processing unit (CPU), a memory, a network, and a disk belong to the resources included in the current system, and after obtaining the usage of the resources, the weights of the resources may be weighted. And, the weighted sum is equivalent to the score of the current system load. For example, if the current system's load score is represented by L, then: where ^ and ^ respectively represent the CPU score weight, the memory score weight, and the network score weight. And disk scoring weights, and u cpu u m t ^ and respectively indicate CPU usage, memory usage, network usage, and disk usage. If the load of the system can be divided into low load, medium load, high load and saturation according to the level from low to high, the current system load can be assigned to the corresponding level according to the system load score L from low to high. , for example, if the load on the system When the score L is 100, the load of the system can be scored in the range of 0 ~ 25, 25 - 50, 50 ~ 75 and 75 ~ 100, and the current system load level is sequentially assigned to the low load, medium load, High load and saturation.
作为获取待合并任务的紧急程度等级的一个实施例,可以将影响所述待合 并任务的紧急程度的因素按照权重进行线性加权求和,根据所述线性加权求和 所得的值对所述待合并任务的紧急程度相应的等级。在本发明实施例中,影响 待合并任务的紧急程度的因素包括但不限于子表中的文件大小、子表中文件个 数和子表中最老数据的超期时间等等。 由于不同的影响因素存在量化的差异, 因此可以采用线性加权求和的方式来计算待合并任务的紧急程度评分。在这种 方式下,对影响待合并任务的紧急程度的每个因素,根据每个因素的最大值计 算出百分比并加权求和, 如下所示, 若待合并任务的紧急程度评分使用 As an embodiment of obtaining the urgency level of the task to be merged, the factors affecting the urgency of the task to be merged may be linearly weighted and summed according to the weight, and the value to be merged according to the value obtained by the linear weighted summation The urgency of the task corresponds to the level. In the embodiment of the present invention, factors affecting the urgency of the task to be merged include, but are not limited to, the file size in the sub-table, the number of files in the sub-table, and the overdue time of the oldest data in the sub-table. Since there are quantitative differences in different influencing factors, the urgency score of the tasks to be combined can be calculated by linear weighted summation. In this way, for each factor affecting the urgency of the task to be combined, the percentage is calculated based on the maximum value of each factor and weighted and summed, as shown below, if the urgency score of the task to be combined is used
ScoreiX^X^...^,)^^, 贝1 J: ScoreiX^X^...^,)^^, Bay 1 J:
Score{Xl,X2,...,Xk) = jwk{Xk I MAXXk)^lQQ% 式( 2 ) 其中, ^表示第 个影响待合并任务的紧急程度的因素, ^表示第 个影 响待合并任务的紧急程度的因素的权重, 而 M X 表示第 个影响待合并任务 的紧急程度的因素能够达到或者设定的最大值。若待合并任务的紧急程度等级 由低到高可依次分为非紧急、普通和紧急等等几种, 则可以按照待合并任务的 紧急程度评分 ^^( ^,...,^)从低到高将待合并任务的紧急程度依次赋予相 应的等级, 例如, 若待合并任务的紧急程度评分 ^ΟΓ ,Χ ,...,^)满分为 100, 则可以将待合并任务的紧急程度评分 1¾^( 1, 2,..., ^)在0 ~ 30、 30 ~ 60和 60 ~ 100三个区间时, 相应地依次将待合并任务的紧急程度等级赋予非紧急、 普通 和紧急。 Score{X l , X 2 ,...,X k ) = j w k {X k I MAX Xk )^lQQ% where (2) where ^ represents the first factor affecting the urgency of the task to be merged, ^ Indicates the weight of the first factor affecting the urgency of the task to be merged, and MX indicates the maximum value that the first factor affecting the urgency of the task to be merged can reach or set. If the urgency level of the task to be merged can be divided into non-emergency, normal and emergency, etc., it can be scored according to the urgency of the task to be merged ^^(^,...,^) The urgency of the task to be merged is assigned to the corresponding level in turn, for example, if the urgency score of the task to be combined is ΟΓ, Χ, ..., ^) is 100, the urgency degree of the task to be combined can be scored. 1 3⁄4^( 1 , 2 ,..., ^) In the interval of 0 ~ 30, 30 ~ 60 and 60 ~ 100, the urgency level of the task to be merged is sequentially assigned to non-emergency, normal and emergency.
按照式(2)对待合并任务的紧急程度进行评分, 考虑的是普通场景下各 个因素的一般状态时, 然而, 在有些场景下, 尽管其他影响待合并任务的紧急 程度的因素的分值可能还比较低,但某个影响待合并任务的紧急程度的因素已 经接近或达到最大值时,其对所述待合并任务的执行合并的迫切程度整体影响 开始呈非线性地增大。 由于式(2)对待合并任务的紧急程度进行评分采用的 是线性加权求和,在某个影响待合并任务的紧急程度的因素已经接近或达到最 大值时, 如果仍然按照式(2)对待合并任务的紧急程度进行评分, 其得到的 评分结果不能如实反映所述待合并任务需要执行合并的紧急程度, 显然不合 理。 According to formula (2), the urgency of the combined task is scored, considering the general state of each factor in the common scenario. However, in some scenarios, the scores of other factors affecting the urgency of the task to be merged may still be If the factors affecting the urgency of the task to be merged are close to or reach the maximum value, the overall impact of the urgency of the execution combination of the tasks to be merged begins to increase non-linearly. Due to the urgency of the formula (2) to treat the combined task Is a linear weighted summation. When the factor affecting the urgency of the task to be merged is close to or reaches the maximum value, if the urgency of the combined task is still scored according to formula (2), the scored result cannot be faithfully reflected. The urgency of the merger to be merged is obviously unreasonable.
本发明实施例提供一种单项影响极值放大加权求和的评分方法,以在个別 影响待合并任务的紧急程度达到或者接近最大值时,能够非线性放大对待合并 任务的紧急程度的评分, 即, 若当前影响所述待合并任务的紧急程度的各因素 中某个因素 的影响值已达到或接近该因素的最大值,则将影响所述待合并任 务的紧急程度的各因素按照权重进行非线性加权求和,根据所述非线性加权求 和所得的值对所述待合并任务的紧急程度赋予相应的等级。具体地,将影响所 述待合并任务的紧急程度的各因素按照权重进行非线性加权求和包括:将影响 所述待合并任务的紧急程度的因素的权重^放大至 , 以所述 作 为影响所述待合并任务的紧急程度的因素新的权重,将影响所述待合并任务的 紧急程度的因素按照所述新的权重进行非线性加权求和,根据所述非线性加权 求和所得的值对所述待合并任务的紧急程度相应的等级,即待合并任务的紧急 程度评分 ^/^:^,:^,…,:^;!表示为:  The embodiment of the invention provides a scoring method for a single item affecting the extreme value amplification weighted summation, so as to achieve a non-linear enlargement of the urgency degree of the combined task when the urgency affecting the task to be combined individually reaches or approaches the maximum value. That is, if the influence value of one of the factors affecting the urgency of the task to be merged has reached or is close to the maximum value of the factor, the factors affecting the urgency of the task to be merged are weighted according to the weight. The nonlinear weighted summation assigns a corresponding level to the urgency of the task to be merged according to the value obtained by the nonlinear weighted summation. Specifically, the nonlinear weighted summation of each factor affecting the urgency of the task to be merged according to the weight includes: enlarging the weight of the factor affecting the urgency of the task to be merged to a new weight for the urgency of the combined task, a factor that affects the urgency of the task to be merged, a nonlinear weighted summation according to the new weight, and a value pair obtained by the nonlinear weighted summation The urgency level of the task to be merged is the urgency score of the task to be merged ^/^:^,:^,...,:^;! Expressed as:
Score{Xl,X2,...,Xk) = jf{Ak)wk{Xkl ΜΑΧΧι ) * 100% 式(3 ) 其中, ^表示第 个影响待合并任务的紧急程度的因素, ^表示第 个影 响待合并任务的紧急程度的因素的权重, M X 表示第 个影响待合并任务, 而 为权重放大函数, 可以表示为: Score{X l , X 2 ,...,X k ) = j f{A k )w k {X k l ΜΑΧ Χι ) * 100% where (3) where ^ denotes the first emergency affecting the task to be merged The degree factor, ^ represents the weight of the first factor affecting the urgency of the task to be merged, MX represents the first task to be merged, and the weight amplification function, which can be expressed as:
f(Ak)=MAXXt /(MAXXt -Xk+AXk) 式(4) 上述 的表达式中, MAXXt表示第 个影响待合并任务的紧急程度的因 素影响值的最大值, ΔΧλ是第 个影响待合并任务的紧急程度的因素的一个极 小值, 用于调节式(4) 以使式(4) 的分母不为 0。 f(A k )=MAX Xt /(MAX Xt -X k +AX k ) Equation (4) In the above expression, MAX Xt represents the maximum value of the influence factor of the first urgency affecting the task to be combined, ΔΧ λ is a minimum value of the first factor affecting the urgency of the task to be merged, and is used to adjust equation (4) such that the denominator of equation (4) is not zero.
需要说明的是, 上述按照式(1)计算出来的评分获取的当前系统的负载 等级, 以及按照式(2)或式(3)计算出来的评分获取的影响待合并任务的紧 急程度等级, 仅仅是举例说明, 并不限于上述式(1)、 式(2)或式(3)的方 式。 It should be noted that the load level of the current system obtained by the above-mentioned score calculated according to the formula (1), and the urgency level of the task to be combined obtained by the score calculated according to the formula (2) or the formula (3) are only For example, it is not limited to the formula (1), (2) or (3) above. Style.
S202,根据所述当前系统的负载等级和待合并任务的紧急程度等级, 匹配 在本发明实施例中,合并任务执行策略是按照系统的负载等级和待合并任 务的紧急程度等级,预先制定而用于确定是否执行任务合并的一套方案, 其原 则示例性地包括: 当前系统的负载等级越高, 允许执行紧急程度等级越高的待 合并任务, 即允许对紧急程度等级越高的待合并任务执行合并操作,在当前系 统的负载等级为饱和时, 不允许执行任何待合并任务, 即不允许执行任何待合 并任务的合并。 对合并任务执行策略, 也可以解读为: 当确定了待合并任务的 紧急程度等级时, 当前系统的负载等级越高, 所述待合并任务允许执行合并的 可能性越低, 反之, 当前系统的负载等级越低, 所述待合并任务允许执行合并 的可能性越高; 或者, 当确定了当前系统的负载等级时, 待合并任务的紧急程 度等级越高 (即待合并任务需要执行合并的程度越紧急), 所述待合并任务允 许执行合并的可能性越高, 反之, 待合并任务的紧急程度等级越低(即待合并 任务需要执行合并的程度越弛緩), 所述待合并任务不允许执行合并的可能性 越高;在当前系统的负载等级最低时,例如, 当前系统的负载等级为低负载时, 可以对任何紧急程度等级的待合并任务执行合并;在当前系统的负载等级最高 时, 例如, 当前系统的负载等级为饱和时, 不允许对任何紧急程度等级的待合 并任务执行合并; 在待合并任务的紧急程度等级最高时, 例如, 待合并任务的 紧急程度等级为紧急时, 除非当前系统的负载等级为最高, 例如, 当前系统的 负载等级为饱和, 否则,都允许对所述紧急程度等级为紧急的待合并任务执行 合并。  S202, according to the load level of the current system and the urgency level of the task to be merged, the matching is performed in the embodiment of the present invention, and the combined task execution strategy is pre-defined according to the load level of the system and the urgency level of the task to be merged. The principle of determining a set of tasks for performing task consolidation includes: The higher the load level of the current system, the higher the urgency level is allowed to perform the task to be merged, that is, the task to be merged is allowed to be higher. The merge operation is performed. When the load level of the current system is saturated, no tasks to be merged are allowed to be executed, that is, the merge of any tasks to be merged is not allowed. The execution strategy of the merged task can also be interpreted as: When the urgency level of the task to be merged is determined, the higher the load level of the current system, the lower the possibility that the task to be merged is allowed to perform the merge, and vice versa, the current system The lower the load level, the higher the probability that the task to be merged is allowed to perform the merge; or the higher the urgency level of the task to be merged when the load level of the current system is determined (ie, the degree to which the task to be merged needs to perform the merge) The more urgent, the higher the probability that the task to be merged is allowed to perform the merge, and the lower the level of urgency of the task to be merged (that is, the more relaxed the degree of the merge to be merged), the task to be merged is not allowed. The higher the probability of performing a merge; when the load level of the current system is the lowest, for example, when the load level of the current system is low load, the merged task of any urgency level can be merged; when the load level of the current system is the highest For example, when the load level of the current system is saturated, no permission is allowed. The urgency level of the tasks to be merged is merged; when the urgency level of the task to be merged is the highest, for example, when the urgency level of the task to be merged is urgent, unless the load level of the current system is the highest, for example, the load of the current system The level is saturated, otherwise, the merging is allowed for the task to be merged with the urgency level being urgent.
按照如前所述, 当前系统的负载等级包括但不限于低负载、 中负载、 高负 载和饱和等几种以及待合并任务的紧急程度等级包括但不限于非紧急、普通和 紧急等几种的实施例, 本发明实施例提供的合并任务执行策略可以如下述表 1 所示:
Figure imgf000010_0001
As mentioned earlier, the current system's load levels include, but are not limited to, low load, medium load, high load, and saturation, and the urgency levels of the tasks to be combined include, but are not limited to, non-emergency, normal, and emergency. For example, the merge task execution policy provided by the embodiment of the present invention may be as shown in Table 1 below:
Figure imgf000010_0001
表 1  Table 1
在上述表 1中, "负载等级"是前述实施例中 "当前系统的负载等级" 的筒 称, 包括 "低负载"、 " 中负载"、 " 高负载" 和 "饱和", "紧急程度等级" 是 前述实施例中 "待合并任务的紧急程度等级" 的筒称, 包括 " 非紧急"、 "普 通" 和 " 紧急"。 "负载等级" 和 "紧急程度等级" 交叉处的项目即表示是否 执行待合并任务的合并, 例如, "负载等级" 为 "低负载" 且 "紧急程度等级" 为 "非紧急" 时, 则可以执行待合并任务的合并。 上述表 1所反映的合并任务 执行策略一般性原则是,对于紧急程度等级为非紧急的待合并任务,在系统的 负载等级为中负载或中负载以上时, 不允许对该待合并业务执行合并, 而是将 该待合并任务返回队列等待; 对于紧急程度等级为普通的待合并任务,在系统 的负载等级为高负载或高负载以上时, 不允许对该待合并业务执行合并, 而是 将该待合并任务返回队列等待; 对于紧急程度等级为紧急的待合并任务, 只要 系统的负载等级并非饱和, 即当前系统的负载等级为低负载、 中负载或高负载 时, 则允许对该待合并业务执行合并。根据所述当前系统的负载等级和待合并 任务的紧急程度等级,匹配合并任务执行策略以确定是否允许执行所述待合并 任务包括: 若匹配到所述待合并任务的紧急程度等级为紧急,在所述当前系统 的负载等级为低负载、 中负载或高负载时, 则确定允许执行所述待合并任务, 即确定允许对所述待合并任务执行合并;若匹配到所述待合并任务的紧急程度 等级为非紧急,在所述当前系统的负载等级为中负载或中负载以上时, 则确定 不允许执行所述待合并任务, 即确定不允许对所述待合并任务执行合并; 若匹 配到所述待合并任务的紧急程度等级为普通,在所述当前系统的负载等级为高 负载或高负载以上时, 则确定不允许执行所述待合并任务, 即确定不允许对所 述待合并任务执行合并。 In the above Table 1, the "load level" is the cartridge of the "current system load level" in the foregoing embodiment, including "low load", "medium load", "high load" and "saturated", "urgency level". " is the nickname of the urgency level of the task to be merged in the foregoing embodiment, including "non-emergency", "normal", and "emergency". The item at the intersection of "load level" and "urgency level" indicates whether to perform the combination of the tasks to be merged. For example, when the "load level" is "low load" and the "urgency level" is "non-emergency", Perform a merge of the tasks to be merged. The general principle of the combined task execution strategy reflected in Table 1 above is that for a task to be merged with a urgency level of non-emergency, when the load level of the system is above the medium load or the medium load, the merged service is not allowed to be merged. , the task to be merged is returned to the queue and waits; for the task to be merged with the urgency level is normal, when the load level of the system is high load or high load, the merged service is not allowed to be merged, but The to-be-consolidated task returns to the queue to wait; for the to-be-consolidated task whose urgency level is urgent, as long as the load level of the system is not saturated, that is, when the current system load level is low load, medium load or high load, the to-be-committed is allowed. Business execution merges. And according to the load level of the current system and the urgency level of the task to be merged, matching the merge task execution policy to determine whether to allow the task to be merged includes: if the urgency level matching the task to be merged is urgent, When the load level of the current system is low load, medium load or high load, it is determined that the task to be merged is allowed to be executed, that is, it is determined that the merge is allowed to be performed on the task to be merged; if the task is to be merged to the task to be merged If the level of the current system is non-emergency, when the load level of the current system is above the medium load or the medium load, it is determined that the task to be merged is not allowed to be executed, that is, it is determined that the merged task is not allowed to be merged; The urgency level assigned to the task to be merged is normal. When the load level of the current system is high load or high load, it is determined that the task to be merged is not allowed to be executed, that is, the waiting for the waiting is not allowed. The merge task performs the merge.
对于表 1示例的合并任务执行策略, 若步骤 S201获取的当前系统的负载等 级为 "低负载" 时, 则匹配到表 1示例的合并任务执行策略, 无论步骤 S201获 取的待合并任务的紧急程度等级是 "非紧急"、 "普通" 还是 "紧急", 都确定 可以执行所述待合并任务的合并; 若步骤 S201获取的当前系统的负载等级为 "中负载", 则匹配到表 1示例的合并任务执行策略, 当步骤 S201获取的待合并 任务的紧急程度等级是 "普通" 或者 "紧急" 时, 也确定可以执行所述待合 并任务的合并; 若步骤 S201获取的当前系统的负载等级为 "高负载", 则匹配 到表 1示例的合并任务执行策略, 只有当步骤 S201获取的待合并任务的紧急程 度等级是 "紧急" 时, 才确定可以执行所述待合并任务的合并; 若步骤 S201 获取的当前系统的负载等级为 "饱和" 时, 则匹配到表 1示例的合并任务执行 策略, 无论步骤 S201获取的待合并任务的紧急程度等级是 "非紧急" "普通" 还是 "紧急", 都确定不可以执行所述待合并任务的合并; 若步骤 S201获取的 当前系统的负载等级为 "高负载", 则匹配到表 1示例的合并任务执行策略, 当 步骤 S201获取的待合并任务的紧急程度等级是 "非紧急" 或者 "普通" 时, 也确定不可以执行所述待合并任务的合并;若步骤 S201获取的当前系统的负载 等级为 "中负载", 则匹配到表 1示例的合并任务执行策略, 当步骤 S201获取的 待合并任务的紧急程度等级是 "非紧急" 时, 也确定不可以执行所述待合并任 务的合并。  For the merge task execution policy of the example in Table 1, if the load level of the current system acquired in step S201 is "low load", the merge task execution policy of the example of Table 1 is matched, regardless of the urgency of the task to be merged acquired in step S201. Whether the level is "non-emergency", "normal" or "emergency", it is determined that the combination of the tasks to be merged can be performed; if the load level of the current system acquired in step S201 is "medium load", it matches the example of Table 1. The merging task execution policy, when the urgency level of the task to be merged obtained in step S201 is "normal" or "emergency", it is also determined that the merging of the tasks to be merged can be performed; if the load level of the current system acquired in step S201 is "High load", then matching to the merge task execution policy of the example in Table 1, only when the urgency level of the task to be merged acquired in step S201 is "emergency", it is determined that the merge of the task to be merged can be performed; When the load level of the current system acquired by S201 is "saturated", it matches the merge task execution of the example in Table 1. It should be noted that, regardless of whether the urgency level of the task to be merged obtained in step S201 is "non-emergency", "normal" or "emergency", it is determined that the merging of the tasks to be merged cannot be performed; if the load level of the current system acquired in step S201 The "high load" is matched to the merge task execution policy of the example in Table 1. When the urgency level of the task to be merged acquired in step S201 is "non-emergency" or "normal", it is also determined that the to-be-committed may not be executed. If the load level of the current system acquired in step S201 is "medium load", the matching task execution policy of the example in Table 1 is matched, and when the urgency level of the task to be merged acquired in step S201 is "non-emergency" It is also determined that the merging of the tasks to be merged cannot be performed.
S203 ,若确定允许执行所述待合并任务,则将所述待合并任务发送至合并 任务执行模块执行任务合并。  S203. If it is determined that the to-be-committed task is allowed to be executed, send the to-be-consolidated task to the merged task execution module to perform task consolidation.
在本发明实施例中,合并任务执行模块是执行任务合并的功能单元,在子 表服务器中可以存在一个或多个这样的功能单元。当合并任务执行模块存在空 闲即可以执行任务合并时, 则主动向调度器索取任务, 或者主动上报自身已处 于空闲状态, 等待调度器分配任务。 因此, 根据步骤 S201和步骤 S202, —旦确 定可以执行所述待合并任务, 即确定可以对所述待合并任务执行合并, 则将所 述待合并任务发送至合并任务执行模块, 由合并任务执行模块执行任务合并。 若确定不可以执行所述待合并任务, 即确定不可以对所述待合并任务执行合 并, 则将所述待合并任务返回等待队列, 等待合适的时机再从等待队列中取出 以执行任务合并。 In the embodiment of the present invention, the merge task execution module is a functional unit that performs task merge, and one or more such functional units may exist in the child table server. When the merged task execution module is idle, the task can be performed, and the task is actively requested from the scheduler, or the active report is idle, and the scheduler is assigned to the task. Therefore, according to step S201 and step S202, if it is determined that the task to be merged can be executed, that is, it is determined that the merge can be performed on the task to be merged, The merged task is sent to the merge task execution module, and the merge task execution module performs task merge. If it is determined that the to-be-combined task cannot be executed, that is, it is determined that the merged task cannot be merged, the to-be-combined task is returned to the waiting queue, and waits for an appropriate timing to be taken out from the waiting queue to perform task combining.
在本发明实施例中, 以当前系统的负载所处的等级,有可能在一段比较长 的时间内, 子表服务器都不宜执行任务合并, 例如, 若当前系统的负载在一段 较长的时间内会处于 "饱和"这一等级, 则按照表 1示例的合并任务执行策略, 无论待合并任务的紧急程度等级是 "非紧急"、 "普通" 还是 "紧急", 都不可 以执行所述待合并任务的合并。如果在上述示例的一段时间内,仍然频繁地从 等待队列取出之前按照合并任务执行策略不能执行任务合并而返回等待队列 中的任务, 显然是一种浪费资源的行为。  In the embodiment of the present invention, it is possible that the sub-table server is not suitable for performing task consolidation for a relatively long period of time, for example, if the load of the current system is in a long period of time. Will be at the level of "saturation", then according to the combined task execution strategy as exemplified in Table 1, no matter whether the urgency level of the task to be merged is "non-emergency", "normal" or "emergency", the to-be-committed can not be executed. The merging of tasks. If, during the period of the above example, the tasks in the waiting queue cannot be performed according to the merge task execution policy before the queue is fetched frequently from the waiting queue, it is obviously a waste of resources.
因此,为了避免在一段比较长的时间内子表服务器都不宜执行任务合并而 仍然频繁地从等待队列中取出待合并任务去执行任务合并, 在本发明实施例 中, 将在步骤 S201和步骤 S202中按照表 1示例的合并任务执行策略不能执行任 务合并而返回等待队列中的待合并任务返回等待队列后,可以将所述待合并任 务冻结,以使所述待合并任务在设定的冻结时间内处于不能取出所述等待队列 的状态。 至于冻结时间设定的大小, 可以根据当前系统的负载等级和 /或所述 待合并任务的紧急程度等级预期持续时间的长短而由用户设定,本发明对此不 做特別限定。  Therefore, in order to prevent the sub-table server from performing the task merging for a relatively long period of time, the task to be merged is still frequently taken out from the waiting queue to perform the task merging. In the embodiment of the present invention, in step S201 and step S202 After the merge task execution policy according to the example in Table 1 cannot perform the task merge and returns to the waiting queue in the waiting queue, the task to be merged may be frozen, so that the task to be merged is within the set freeze time. It is in a state in which the waiting queue cannot be taken out. The size of the freeze time setting may be set by the user according to the load level of the current system and/or the expected duration of the urgency level of the task to be merged, which is not particularly limited in the present invention.
在本发明实施例中, 当将所述待合并任务冻结后, 还可以启动一计时器, 对所述设定的冻结时间进行计时, 以使在所述设定的冻结时间结束后,将在步 骤 S201和步骤 S202中按照表 1示例的合并任务执行策略不能执行任务合并而返 回等待队列中的待合并任务更改为能够取出等待队列的状态,从而重新评估所 述待合并任务是否可以执行合并。  In the embodiment of the present invention, after the task to be merged is frozen, a timer may be started to time the set freeze time so that after the set freeze time is over, In step S201 and step S202, the merge task execution policy according to the table 1 cannot perform the task merge and the task to be merged in the wait queue is changed to the state in which the wait queue can be taken out, thereby reevaluating whether the merge task can perform the merge.
由于冻结时间是基于当前系统的负载所处等级的预期持续时间的长短而 设定, 因此,也不能排除冻结时间的设定超过当前系统的负载所处等级的预期 持续时间。 例如, 若当前系统的负载等级或 /和所述待合并任务的紧急程度等 级出现变化,从而按照表 1示例的合并任务执行策略,对之前按照表 1示例的合 并任务执行策略不能执行合并的待合并任务可以重新执行合并, 此时,如果机 械地等待设定的冻结时间结束才执行任务合并,对系统而言也是一种资源的浪 费。 Since the freeze time is set based on the length of the expected duration of the current system's load level, it is also impossible to exclude that the freeze time setting exceeds the expected duration of the current system load level. For example, if the load level of the current system or/and the urgency level of the task to be merged changes, thereby performing the merge task execution strategy as exemplified in Table 1, the previous example according to Table 1 And the task execution strategy can not perform the merged to-be-merge task, and the merge can be re-executed. At this time, if the mechanical merge is waited for the set freeze time to end, the task merge is performed, which is also a waste of resources for the system.
因此, 在本发明实施例中, 对于当前系统的负载等级发生的变化, 即, 若 所述当前系统的负载等级由 "饱和"下降为 "高负载"、 "中负载"或 "低负载", 则可以将在步骤 S201和步骤 S202中按照表 1示例的合并任务执行策略不能执行 任务合并而返回等待队列中的待合并任务更改为能够取出等待队列的状态;或 者, 若所述当前系统的负载等级由 "高负载" 下降为 "中负载" 或 "低负载", 则可以将在步骤 S201和步骤 S202中按照表 1示例的合并任务执行策略不能执行 任务合并而返回等待队列中的待合并任务更改为能够取出等待队列的状态;或 者, 若所述当前系统的负载等级由 "中负载" 下降为 "低负载", 则可以将在 步骤 S201和步骤 S202中按照表 1示例的合并任务执行策略不能执行任务合并而 返回等待队列中的待合并任务更改为能够取出等待队列的状态。  Therefore, in the embodiment of the present invention, a change occurs in the load level of the current system, that is, if the load level of the current system is decreased from "saturated" to "high load", "medium load" or "low load", Then, in step S201 and step S202, the merge task execution policy according to the table 1 may not perform task merge, and the task to be merged in the wait queue is changed to a state in which the wait queue can be taken out; or, if the load of the current system is If the level is lowered from "high load" to "medium load" or "low load", the merge task execution strategy according to Table 1 in step S201 and step S202 can not perform task merge and return to the task to be merged in the wait queue. Change to a state in which the waiting queue can be taken out; or, if the load level of the current system is decreased from "medium load" to "low load", the merge task execution strategy according to Table 1 in steps S201 and S202 can be executed. The task to be merged and the task to be merged in the waiting queue cannot be changed to the state in which the waiting queue can be taken out.
在本发明另一实施例中,对于待合并任务的紧急程度等级发生的变化, 即 若在步骤 S201和步骤 S202中按照表 1示例的合并任务执行策略不能执行任务合 并而返回等待队列中的待合并任务的紧急程度等级由 "非紧急"上升为 "普通" 或 "紧急" , 则将所述待合并任务更改为能够取出等待队列的状态; 或者, 若在步骤 S201和步骤 S202中按照表 1示例的合并任务执行策略不能执行任务合 并而返回等待队列中的待合并任务的紧急程度等级由 "普通"上升为 "紧急", 则将所述待合并任务更改为能够取出等待队列的状态。  In another embodiment of the present invention, the change of the urgency level of the task to be merged, that is, if the merged task execution policy according to the table 1 in step S201 and step S202 cannot perform the task merging and returns to the waiting queue If the urgency level of the merging task is raised from "non-emergency" to "normal" or "emergency", the task to be merged is changed to a state in which the waiting queue can be taken out; or, if in step S201 and step S202, according to Table 1 The example merge task execution policy cannot perform task merge and the urgency level of the task to be merged in the wait queue is raised from "normal" to "emergency", and the task to be merged is changed to a state in which the wait queue can be taken out.
作为本发明提前结束冻结时间的又一实施例,在等待队列中所有待合并任 务均冻结并且合并任务执行模块处于空闲状态时, 若在步骤 S201和步骤 S202 中按照表 1示例的合并任务执行策略不能执行任务合并而返回等待队列中的待 合并任务的紧急程度等级比其他待合并任务的紧急程度等级高,则将所述在步 骤 S201和步骤 S202中按照表 1示例的合并任务执行策略不能执行任务合并而返 回等待队列中的待合并任务更改为能够取出所述等待队列的状态。  As another embodiment of the present invention for ending the freeze time in advance, when all the tasks to be merged in the waiting queue are frozen and the merge task execution module is in an idle state, if the merge task execution strategy according to Table 1 is performed in steps S201 and S202 If the urgency level of the task to be merged in the waiting queue cannot be performed and the urgency level of the task to be merged is higher than that of the other tasks to be merged, the merged task execution policy according to the example in Table 1 in step S201 and step S202 cannot be executed. The tasks to be merged and returned to the waiting queue are changed to a state in which the waiting queue can be taken out.
从上述本发明实施例提供的键值数据库的数据合并方法可知,由于待合并 任务是否执行合并是根据当前系统的负载等级和待合并任务的紧急程度等级, 匹配合并任务执行策略来确定。因此,与现有技术不考虑系统的负载或状态(例 如, 系统比较繁忙和 /或系统负载比较高) 而仍然无限制地执行待合并任务的 合并相比,本发明实施例提供的方法能够根据系统资源占用情况来调度数据合 并任务,让数据合并任务在系统负载高时自动延迟,在系统低负载时自动执行, 从而达到减小执行任务合并对访问业务影响的同时,又能够让待合并任务在合 适的时机自动、 及时地执行合并。 According to the data merging method of the key value database provided by the embodiment of the present invention, whether the merging of the task to be merged is based on the load level of the current system and the urgency level of the task to be merged, Match the merge task execution strategy to determine. Therefore, the method provided by the embodiment of the present invention can be based on the prior art without considering the load or the state of the system (for example, the system is relatively busy and/or the system load is relatively high) and the merge of the tasks to be merged is still performed without limitation. The system resource occupation situation is used to schedule the data merge task, so that the data merge task is automatically delayed when the system load is high, and is automatically executed when the system is under low load, thereby reducing the impact of the execution task merge on the access service, and allowing the task to be merged. The merger is performed automatically and in a timely manner at the right time.
请参阅附图 3 , 是本发明实施例提供的键值数据库的数据合并装置结构示 意图。 为了便于说明, 仅仅示出了与本发明实施例相关的部分。 附图 3示例的 键值数据库的数据合并装置包括获取模块 301、 确定模块 302和处理模块 303 , 其中:  Referring to FIG. 3, it is a structural diagram of a data merging device of a key value database according to an embodiment of the present invention. For the convenience of description, only parts related to the embodiment of the present invention are shown. The data combining apparatus of the key value database illustrated in Fig. 3 includes an obtaining module 301, a determining module 302, and a processing module 303, wherein:
获取模块 301 , 用于获取当前系统的负载等级和待合并任务的紧急程度等 级。  The obtaining module 301 is configured to acquire a load level of the current system and an urgency level of the task to be merged.
由于键值数据库的合并任务是否执行与当前系统的负载有关, 例如, 在系 统比较繁忙和 /或系统负载比较高时, 无限制地执行键值数据库的合并任务会 加重系统负担, 影响数据访问业务的性能。 除此之外, 键值数据库的合并任务 是否执行还与待合并任务的紧急程度有关。 因此, 在本实施例中, 可以首先由 等级获取模块 301获取当前系统的负载等级和待合并任务的紧急程度等级, 其 中, 当前系统的负载等级包括但不限于低负载、中负载、高负载和饱和等几种, 而待合并任务的紧急程度等级包括但不限于非紧急、 普通和紧急等几种。  Whether the merge task of the key value database is related to the load of the current system, for example, when the system is busy and/or the system load is relatively high, the unrestricted execution of the key database merge task will increase the system burden and affect the data access service. Performance. In addition to this, whether or not the merge task of the key-value database is executed is also related to the urgency of the task to be merged. Therefore, in this embodiment, the level of the current system and the level of urgency of the task to be merged may be first acquired by the level obtaining module 301, where the load level of the current system includes but is not limited to low load, medium load, high load, and There are several types of saturation, such as non-emergency, normal and emergency.
确定模块 302, 用于根据所述当前系统的负载等级和待合并任务的紧急程 在本实施例中,合并任务执行策略是按照系统的负载等级和待合并任务的 紧急程度等级,预先制定而用于确定是否执行任务合并的一套方案, 其原则示 例性地包括:当前系统的负载等级越高,允许执行紧急程度越高的待合并任务, 即允许对紧急程度等级越高的待合并任务执行合并,在当前系统的负载等级为 饱和时, 不允许执行任何待合并任务, 即不允许执行任何待合并任务的合并。 在本发明实施例中, 对合并任务执行策略, 也可以解读为: 当确定了待合并任 务的紧急程度等级时, 当前系统的负载等级越高, 所述待合并任务允许执行合 并的可能性越低, 反之, 当前系统的负载等级越低, 所述待合并任务允许执行 合并的可能性越高; 或者, 当确定了当前系统的负载等级时, 待合并任务的紧 急程度等级越高 (即待合并任务需要执行合并的程度越紧急 ), 所述待合并任 务允许执行合并的可能性越高, 反之, 待合并任务的紧急程度等级越低(即待 合并任务需要执行合并的程度越弛緩), 所述待合并任务不允许执行合并的可 能性越高; 在当前系统的负载等级最低时, 例如, 当前系统的负载等级为低负 载时, 可以对任何紧急程度等级的待合并任务执行合并; 在当前系统的负载等 级最高时, 例如, 当前系统的负载等级为饱和时, 不允许对任何紧急程度等级 的待合并任务执行合并; 在待合并任务的紧急程度等级最高时, 例如, 待合并 任务的紧急程度等级为紧急时, 除非当前系统的负载等级为最高, 例如, 当前 系统的负载等级为饱和, 否则,都允许对所述紧急程度等级为紧急的待合并任 务执行合并。 The determining module 302 is configured to use, according to the load level of the current system and the emergency process of the task to be merged, in the embodiment, the combined task execution policy is pre-defined according to the load level of the system and the urgency level of the task to be merged. The principle of determining whether to perform task consolidation includes the following examples: the higher the load level of the current system, the higher the urgency task to be merged, that is, the higher the urgency level is to be performed. Merging, when the current system's load level is saturated, it is not allowed to perform any tasks to be merged, that is, it is not allowed to perform the merge of any tasks to be merged. In the embodiment of the present invention, the execution policy of the merged task may also be interpreted as: when the urgency level of the task to be merged is determined, the higher the load level of the current system, the task to be merged is allowed to execute. The lower the probability of the sum, the lower the load level of the current system, the higher the probability that the task to be merged is allowed to perform the merge; or the urgency level of the task to be merged when the load level of the current system is determined The higher the degree (ie, the more urgent the task to be merged needs to perform the merge), the higher the probability that the task to be merged is allowed to perform the merge, and vice versa, the lower the level of urgency of the task to be merged (ie, the task to be merged needs to perform the merge) The more relaxed the degree, the higher the probability that the task to be merged is not allowed to perform the merge; when the load level of the current system is the lowest, for example, when the load level of the current system is low load, the urgency level can be merged. Task execution merge; When the current system's load level is the highest, for example, when the current system's load level is saturated, it is not allowed to perform merging for any urgency level to be merged task; when the urgency level of the task to be merged is the highest, for example , the urgency level of the task to be merged is urgent, unless the current system The load level is the highest, for example, the load level of the current system is saturated, otherwise, the merge is allowed to be performed on the task to be merged whose emergency level is urgent.
按照如前所述, 当前系统的负载等级包括但不限于低负载、 中负载、 高负 载和饱和等几种以及待合并任务的紧急程度等级包括但不限于非紧急、普通和 紧急等几种的实施例,本实施例提供的合并任务执行策略可以如前述表 1所示。  As mentioned earlier, the current system's load levels include, but are not limited to, low load, medium load, high load, and saturation, and the urgency levels of the tasks to be combined include, but are not limited to, non-emergency, normal, and emergency. For example, the merge task execution policy provided in this embodiment may be as shown in Table 1 above.
处理模块 303 , 用于若所述确定模块 302确定执行所述待合并任务, 则将所 述待合并任务发送至合并任务执行模块执行任务合并。  The processing module 303 is configured to: if the determining module 302 determines to perform the to-be-committed task, send the to-be-combined task to the merged task execution module to perform task combining.
在本实施例中,合并任务执行模块是执行任务合并的功能单元,在子表服 务器中可以存在一个或多个这样的功能单元。当合并任务执行模块存在空闲即 可以执行任务合并时, 则主动向调度器索取任务, 或者主动上报自身已处于空 闲状态, 等待调度器分配任务。 因此, 一旦确定模块 302确定可以执行所述待 合并任务, 即确定可以对所述待合并任务执行合并, 则处理模块 303将所述待 合并任务发送至合并任务执行模块, 由合并任务执行模块执行任务合并。若确 定模块 302确定不可以执行所述待合并任务, 即确定不可以对所述待合并任务 执行合并, 则处理模块 303还可以将所述待合并任务返回等待队列, 等待合适 的时机再从等待队列中取出以执行任务合并。  In this embodiment, the merge task execution module is a functional unit that performs task merge, and one or more such functional units may exist in the child table server. When the merged task execution module is idle, the task can be performed, and the task is actively requested from the scheduler, or the active report is idle, and the scheduler is assigned to the task. Therefore, once the determining module 302 determines that the to-be-combined task can be performed, that is, it is determined that the merge can be performed on the to-be-combined task, the processing module 303 sends the to-be-combined task to the merged task execution module, which is executed by the merged task execution module. Task consolidation. If the determining module 302 determines that the to-be-combined task cannot be performed, that is, it is determined that the merged task cannot be merged, the processing module 303 may further return the to-be-combined task to the waiting queue, wait for a suitable timing, and wait for another time. Take out the queue to perform task consolidation.
需要说明的是, 以上键值数据库的数据合并装置的实施方式中,各功能模 块的划分仅是举例说明, 实际应用中可以根据需要, 例如相应硬件的配置要求 或者软件的实现的便利考虑, 而将上述功能分配由不同的功能模块完成, 即将 所述键值数据库的数据合并装置的内部结构划分成不同的功能模块,以完成以 上描述的全部或者部分功能。 而且, 实际应用中, 本实施例中的相应的功能模 块可以是由相应的硬件实现,也可以由相应的硬件执行相应的软件完成,例如, 前述的等级获取模块,可以是具有执行前述获取当前系统的负载等级和待合并 任务的紧急程度等级的硬件, 例如等级获取器,也可以是能够执行相应计算机 程序从而完成前述功能的一般处理器或者其他硬件设备; 再如前述的确定模 块,可以是具有执行前述根据所述当前系统的负载等级和待合并任务的紧急程 件, 例如确定器,也可以是能够执行相应计算机程序从而完成前述功能的一般 处理器或者其他硬件设备(本说明书提供的各个实施例都可应用上述描述原 则)。 It should be noted that, in the implementation manner of the data merging device of the above key value database, the division of each functional module is only an example, and the actual application may be configured according to requirements, for example, corresponding hardware requirements. Or the convenience of the implementation of the software, and the above-mentioned function allocation is completed by different functional modules, that is, the internal structure of the data combining device of the key value database is divided into different functional modules to complete all or part of the functions described above. Moreover, in practical applications, the corresponding functional modules in this embodiment may be implemented by corresponding hardware, or may be executed by corresponding hardware. For example, the foregoing level obtaining module may have the foregoing acquisition current The hardware of the load level of the system and the urgency level of the task to be merged, such as a level acquirer, may also be a general processor or other hardware device capable of executing a corresponding computer program to perform the aforementioned functions; An emergency program, such as a determiner, that performs the foregoing load level according to the current system and a task to be merged, such as a determiner, may also be a general processor or other hardware device capable of executing a corresponding computer program to perform the aforementioned functions (each provided in this specification) The above described principles can be applied to the embodiments).
附图 3示例的获取模块 301可以包括第一获取单元 401 ,如附图 4所示本发明 另一实施例提供的键值数据库的数据合并装置。 第一获取单元 401用于将当前 系统包含的资源的使用情况按照权重进行加权求和,根据所述加权求和所得的 值对所述当前系统的负载赋予相应的等级, 其中, 所述系统的资源的使用情况 包括 CPU的使用情况、 内存使用情况、 网络使用情况和磁盘使用情况的任意组 合, 所述当前系统的负载等级包括低负载、 中负载、 高负载和饱和。  The acquisition module 301 of the example of FIG. 3 may include a first acquisition unit 401, such as the data integration device of the key value database provided by another embodiment of the present invention as shown in FIG. The first obtaining unit 401 is configured to perform a weighted summation of the usage of the resources included in the current system according to the weight, and assign a corresponding level to the load of the current system according to the value obtained by the weighted summation, where the system is Resource usage includes any combination of CPU usage, memory usage, network usage, and disk usage, including current load, medium load, high load, and saturation.
在附图 4示例的键值数据库的数据合并装置中, 中央处理器 (Central Processing Unit, CPU ), 内存、 网络和磁盘等都属于当前系统包含的资源。 第 一获取单元 401可以在获取这些资源的使用情况后, 按照各个资源的权重进行 加权求和, 加权求和所得的值相当于是对当前系统的负载进行的评分, 例如, 若当前系统的负载评分使用 L表示, 则 L的表达式如前述式(1 )所示。 若系统 的负载按照等级由低到高可依次分为低负载、 中负载、 高负载和饱和等几种, 则第一获取单元 401可以按照系统的负载评分 L从低到高将当前系统的负载依 次赋予相应的等级, 例如, 若系统的负载评分 L满分为 100, 则将系统的负载评 分 L在 0 ~ 25、 25 - 50、 50 ~ 75和 75 ~ 100四个区间时, 第一获取单元 401相应 地依次将当前系统的负载等级赋予低负载、 中负载、 高负载和饱和。 需要说明的是, 在附图 4示例的键值数据库的数据合并装置中, 其获取模 块 301还可以包括其他功能模块或单元,用于获取待合并任务的紧急程度等级, 其获取方式在附图 4示例的装置中不做限定。 In the data merging device of the key value database illustrated in FIG. 4, a central processing unit (CPU), a memory, a network, a disk, and the like all belong to resources included in the current system. The first obtaining unit 401 may perform weighted summation according to the weight of each resource after acquiring the usage of the resources, and the weighted summation value is equivalent to scoring the current system load, for example, if the current system load score When L is used, the expression of L is as shown in the above formula (1). If the load of the system can be divided into low load, medium load, high load and saturation according to the level from low to high, the first obtaining unit 401 can load the current system according to the load score L of the system from low to high. The corresponding level is assigned in turn. For example, if the load score of the system is 100, the first acquisition unit is when the load score of the system is in the range of 0 to 25, 25 to 50, 50 to 75, and 75 to 100. 401 correspondingly sequentially assigns the load level of the current system to low load, medium load, high load, and saturation. It should be noted that, in the data merging device of the key value database illustrated in FIG. 4, the acquiring module 301 may further include other functional modules or units for acquiring the urgency level of the task to be merged, and the obtaining manner thereof is in the drawing. 4 The device of the example is not limited.
附图 3示例的等级获取模块 301也可以包括第二获取单元 501,如附图 5所示 本发明另一实施例提供的键值数据库的数据合并装置。 第二获取单元 501用于 将影响所述待合并任务的紧急程度的因素按照权重进行线性加权求和,根据所 述线性加权求和所得的值对所述待合并任务的紧急程度赋予相应的等级, 其 中, 所述影响所述待合并任务的紧急程度的各因素包括子表中的文件大小、子 表中文件个数和子表中最老数据的超期时间的任意组合,所述待合并任务的紧 急程度等级包括普通、 紧急和必须。  The level obtaining module 301 of the example of FIG. 3 may also include a second obtaining unit 501, such as the data combining device of the key value database provided by another embodiment of the present invention as shown in FIG. The second obtaining unit 501 is configured to linearly weight the factors affecting the urgency of the task to be merged according to the weight, and assign a corresponding level to the urgency of the task to be merged according to the value obtained by the linear weighted summation. The factors affecting the urgency of the task to be merged include any combination of the file size in the sub-table, the number of files in the sub-table, and the expiration time of the oldest data in the sub-table, where the task to be merged Urgency levels include general, urgent, and mandatory.
在附图 5示例的键值数据库的数据合并装置中, 影响待合并任务的紧急程 度的因素包括但不限于子表中的文件大小、子表中文件个数和子表中最老数据 的超期时间等等。 由于不同的影响因素存在量化的差异, 因此, 第二获取单元 501可以采用线性加权求和的方式来计算待合并任务的紧急程度评分。 在这种 方式下,对影响待合并任务的紧急程度的每个因素,根据每个因素的最大值计 算出百分比并加权求和, 若待合并任务的紧急程度评分 Srar 用前 述式(2 )表示, 待合并任务的紧急程度等级由低到高可依次分为普通、 紧急 和必须等几种, 则第二获取单元 501可以按照待合并任务的紧急程度评分 In the data merging device of the key value database illustrated in FIG. 5, factors affecting the urgency of the task to be merged include, but are not limited to, the file size in the sub-table, the number of files in the sub-table, and the overdue time of the oldest data in the sub-table. and many more. Since there are quantitative differences in different influencing factors, the second obtaining unit 501 can calculate the urgency score of the task to be merged by means of linear weighted summation. In this way, for each factor affecting the urgency of the task to be combined, the percentage is calculated according to the maximum value of each factor and weighted and summed. If the urgency score Srar of the task to be combined is expressed by the above formula (2) The urgency level of the task to be merged may be divided into normal, urgent, and mandatory, and the second obtaining unit 501 may score according to the urgency of the task to be merged.
, 从低到高将待合并任务的紧急程度依次赋予相应的等级, 例 如, 若待合并任务的紧急程度评分
Figure imgf000017_0001
)满分为 100, 则将待合并任 务的紧急程度评分 。 ( 1, 2, ..., ^)在0 ~ 30、 30 ~ 60和 60 ~ 100三个区间时, 第二获取单元 501相应地依次将待合并任务的紧急程度等级赋予普通、 紧急和 必须。
, from low to high, assign the urgency of the task to be merged to the corresponding level, for example, if the urgency score of the task to be merged
Figure imgf000017_0001
A perfect score of 100, the urgency of the task to be combined is scored. ( 1 , 2 , ..., ^) In the interval of 0 to 30, 30 to 60, and 60 to 100, the second obtaining unit 501 sequentially assigns the urgency level of the task to be combined to ordinary, urgent, and necessary .
需要说明的是, 在附图 5示例的键值数据库的数据合并装置中, 其获取模 块 301还可以包括其他功能模块或单元, 用于获取当前系统的负载等级, 其获 取方式在附图 5示例的装置中不做限定。  It should be noted that, in the data merging device of the key value database illustrated in FIG. 5, the obtaining module 301 may further include other functional modules or units for acquiring the load level of the current system, and the obtaining manner thereof is illustrated in FIG. 5. The device is not limited.
按照式(2 )对待合并任务的紧急程度进行评分, 考虑的是普通场景下各 个因素的一般状态时, 然而, 在有些场景下, 尽管其他影响待合并任务的紧急 程度的因素的分值可能还比较低,但某个影响待合并任务的紧急程度的因素已 经接近或达到最大值时,其对所述待合并任务的执行合并的迫切程度整体影响 开始呈非线性地增大。 由于式(2 )对待合并任务的紧急程度进行评分采用的 是线性加权求和在某个影响待合并任务的紧急程度的因素已经接近或达到最 大值时,第二获取单元 501仍然按照式( 2 )对待合并任务的紧急程度进行评分, 其得到的评分结果不能如实反映所述待合并任务的紧急程度,显然不合理。有 必要在个別影响待合并任务的紧急程度达到或者接近最大值时,能够非线性放 大对待合并任务的紧急程度的评分, 即, 若当前影响所述待合并任务的紧急程 度的各因素中因素 的影响值已达到或接近所述因素 的最大值, 则附图 3示 例的获取模块 301也可以包括第三获取单元 601,如附图 6a所示本发明另一实施 例提供的键值数据库的数据合并装置。 According to formula (2), the urgency of the combined task is scored, considering the general state of each factor in the common scenario, however, in some scenarios, despite other emergency affecting the task to be merged The score of the degree factor may still be relatively low, but when the factor affecting the urgency of the task to be merged is close to or reaches the maximum value, the overall impact of the urgency of the execution combination of the task to be merged begins to be nonlinear. The ground is enlarged. Since the urgency of the merging task is evaluated by the formula (2), the linear weighted summation is used. When the factor affecting the urgency of the task to be merged has approached or reached the maximum value, the second obtaining unit 501 still follows the formula (2). The urgency of the combined task is scored, and the scored result cannot accurately reflect the urgency of the task to be merged, which is obviously unreasonable. It is necessary to non-linearly amplify the urgency of the task to be combined when the urgency of the individual task to be merged reaches or approaches the maximum value, that is, if the factors affecting the urgency of the task to be combined currently The acquisition module 301 exemplified in FIG. 3 may also include a third acquisition unit 601, as shown in FIG. 6a, which is a key value database provided by another embodiment of the present invention. Data merge device.
第三获取单元 601 , 用于将影响所述待合并任务的紧急程度的各因素按照 权重进行非线性加权求和,根据所述加权求和所得的值对所述待合并任务的紧 急程度赋予相应的等级,所述影响所述待合并任务的紧急程度的各因素包括子 表中的文件大小、 子表中文件个数和子表中最老数据的超期时间的任意组合, 所述待合并任务的紧急程度等级包括非紧急、 普通和紧急。  a third obtaining unit 601, configured to perform non-linear weighted summation of each factor affecting the urgency of the task to be merged according to a weight, and assign a urgency to the task to be merged according to the value obtained by the weighted summation The level of the urgency affecting the task to be merged includes any combination of the file size in the child table, the number of files in the child table, and the time of the oldest data in the child table, the task to be merged Urgency levels include non-emergency, general and emergency.
需要说明的是, 在附图 6a示例的键值数据库的数据合并装置中, 其获取模 块 301还可以包括其他功能模块或单元, 用于获取当前系统的负载等级, 其获 取方式在附图 6a示例的装置中不做限定。  It should be noted that, in the data merging device of the key value database illustrated in FIG. 6a, the obtaining module 301 may further include other functional modules or units for acquiring the load level of the current system, and the obtaining manner thereof is illustrated in FIG. 6a. The device is not limited.
附图 6a示例的第三获取单元 601可以进一步包括放大单元 6011和求和单元 The third obtaining unit 601 illustrated in Fig. 6a may further include an amplifying unit 6011 and a summing unit
6012,如附图 6b所示本发明另一实施例提供的键值数据库的数据合并装置, 其 中: 6012, as shown in FIG. 6b, a data merging device for a key value database according to another embodiment of the present invention, wherein:
放大单元 6011 ,用于将影响所述待合并任务的紧急程度的因素的权重 ^放 大至 所述 为权重放大函数。  The amplifying unit 6011 is configured to expand the weight of the factor affecting the urgency of the task to be merged to the weight amplifying function.
求和单元 6012, 用于以所述 作为影响所述待合并任务的紧急程度 的因素新的权重,将影响所述待合并任务的紧急程度的因素按照所述新的权重 进行加权求和,根据所述加权求和所得的值对所述待合并任务的紧急程度相应 的等级, 即, 求和单元 6012按照式(3 )对所述待合并任务的紧急程度赋予相 应的等级, 包括普通、 紧急和必须等等。 a summation unit 6012, configured to weight the factors affecting the urgency of the task to be merged according to the new weight according to the new weight as a factor affecting the urgency of the task to be merged, according to the new weight a value corresponding to the urgency of the task to be merged, that is, a summation unit 6012 assigns a urgency to the task to be merged according to formula (3) The level of responsibility, including general, urgent, and must, and so on.
附图 4至附图 6b示例的确定模块 302可以包括第一确定单元 701、 第二确定 单元 702和第三确定单元 703 ,如附图 7a至附图 7d所示本发明另一实施例提供的 键值数据库的数据合并装置, 其中:  The determining module 302 illustrated in FIG. 4 to FIG. 6b may include a first determining unit 701, a second determining unit 702, and a third determining unit 703, as shown in FIG. 7a to FIG. 7d, which are provided by another embodiment of the present invention. A data merging device for a key value database, where:
第一确定单元 701,用于若匹配到所述待合并任务的紧急程度等级为紧急, 在所述当前系统的负载等级为低负载、 中负载或高负载时, 则确定允许执行所 述待合并任务的合并。  The first determining unit 701 is configured to: if the urgency level of the task to be merged is urgent, when the load level of the current system is low load, medium load, or high load, determine to allow execution of the to-be-committed The merging of tasks.
即, 若获取模块 301获取所述待合并任务的紧急程度等级为 "紧急" 时, 则匹配到表 1示例的合并任务执行策略,获取模块 301获取的当前系统的负载等 级是 "低负载", 第一确定单元 701确定允许执行所述待合并任务, 即确定允许 对所述待合并任务执行合并, 或者, 获取模块 301获取的当前系统的负载等级 是 "中负载", 第一确定单元 701确定允许执行所述待合并任务, 即确定允许对 所述待合并任务执行合并, 或者, 获取模块 301获取的当前系统的负载等级是 "高负载", 第一确定单元 701确定允许执行所述待合并任务, 即确定允许对所 述待合并任务执行合并。  That is, if the urgency level of the to-be-combined task is "emergency", the acquisition module 301 matches the merge task execution policy of the example in Table 1. The load level of the current system acquired by the acquisition module 301 is "low load". The first determining unit 701 determines that the task to be merged is allowed to be executed, that is, determines that the merging is allowed to be performed on the task to be merged, or the load level of the current system acquired by the obtaining module 301 is “medium load”, and the first determining unit 701 determines The task to be merged is allowed to be executed, that is, it is determined that the merging is allowed to be performed on the to-be-combined task, or the load level of the current system acquired by the obtaining module 301 is “high load”, and the first determining unit 701 determines that the to-be-committed is allowed to be executed. The task, ie determining to allow the merging of the tasks to be merged.
第二确定单元 702, 用于若匹配到所述待合并任务的紧急程度等级为非紧 急,在所述当前系统的负载等级为中负载或中负载以上时, 则确定不允许执行 所述待合并任务, 即确定不允许对所述待合并任务执行合并。  The second determining unit 702 is configured to: if the urgency level of the task to be merged is non-emergency, when the load level of the current system is greater than the medium load or the medium load, determine that the to-be-committed is not allowed to be performed. The task, that is, determines that the merging of the tasks to be merged is not allowed.
即,若获取模块 301获取 所述待合并任务的紧急程度等级为 "非紧急"时, 则匹配到表 1示例的合并任务执行策略,获取模块 301获取的当前系统的负载等 级是 "中负载", 第二确定单元 702确定不允许执行所述待合并任务, 即确定 不允许对所述待合并任务执行合并, 或者, 获取模块 301获取的当前系统的负 载等级是 "高负载", 第二确定单元 702确定不允许执行所述待合并任务, 即 确定不允许对所述待合并任务执行合并, 或者, 获取模块 301获取的当前系统 的负载等级是 "饱和", 第二确定单元 702确定不允许执行所述待合并任务, 即确定不允许对所述待合并任务执行合并。  That is, if the urgency level of the to-be-combined task is "non-emergency", the acquisition module 301 matches the merge task execution policy of the example in Table 1. The load level of the current system acquired by the acquisition module 301 is "medium load". The second determining unit 702 determines that the to-be-combined task is not allowed to be executed, that is, it is determined that the merging is not allowed to be performed on the to-be-joined task, or the load level of the current system acquired by the obtaining module 301 is “high load”, and the second determination is performed. The unit 702 determines that the task to be merged is not allowed to be executed, that is, it is determined that the merge is not allowed to be performed on the task to be merged, or the load level of the current system acquired by the obtaining module 301 is "saturated", and the second determining unit 702 determines that the task is not allowed. The tasks to be merged are executed, that is, it is determined that the merge is not allowed to be performed on the tasks to be merged.
第三确定单元 703,用于若匹配到所述待合并任务的紧急程度等级为普通, 在所述当前系统的负载等级为高负载或高负载以上时,则确定不允许执行所述 待合并任务。 The third determining unit 703 is configured to: if the urgency level of the task to be merged is normal, when the load level of the current system is higher than a high load or a high load, determine that the performing is not allowed To be merged.
即, 若获取模块 301获取的所述待合并任务的紧急程度等级为 "普通" 时, 则匹配到表 1示例的合并任务执行策略,在获取模块 301获取的所述当前系统的 负载等级是 "高负载" 或 "饱和" 时, 第三确定单元 703确定不允许执行所述 待合并任务, 即确定不允许对所述待合并任务执行合并。  That is, if the urgency level of the task to be merged acquired by the obtaining module 301 is "normal", the merge task execution policy of the example of Table 1 is matched, and the load level of the current system acquired by the obtaining module 301 is " When the high load is "saturated" or "saturated", the third determining unit 703 determines that the to-be-combined task is not allowed to be executed, that is, it is determined that the merging of the to-be-combined task is not allowed.
附图 4至附图 6b示例的键值数据库的数据合并装置可以包括任务冻结模块 801 , 如附图 8a至附图 8d所示本发明另一实施例提供的键值数据库的数据合并 装置。 任务冻结模块 801用于将所述待合并任务冻结, 以使所述待合并任务在 设定的冻结时间内处于不能取出所述等待队列的状态。  The data merging apparatus of the key value database illustrated in Figures 4 through 6b may include a task freeze module 801, such as the data merging apparatus of the key value database provided by another embodiment of the present invention, as shown in Figures 8a to 8d. The task freeze module 801 is configured to freeze the to-be-consolidated task, so that the to-be-combined task is in a state in which the waiting queue cannot be taken out within a set freeze time.
以当前系统的负载所处的等级,有可能在一段比较长的时间内, 子表服务 器都不宜执行任务合并, 例如, 若当前系统的负载在一段较长的时间内会处于 At the level of the current system load, it is possible that the sub-table server should not perform task consolidation for a relatively long period of time, for example, if the current system load is in a long period of time
"饱和" 这一等级, 则按照表 1示例的合并任务执行策略, 无论待合并任务的 紧急程度等级是 "普通"、 "紧急" 还是 "必须", 都不可以执行所述待合并任 务的合并。如果在上述示例的一段时间内,仍然频繁地从等待队列取出之前按 照合并任务执行策略不能执行任务合并而返回等待队列中的任务,显然是一种 浪费资源的行为。 因此, 为了避免在一段比较长的时间内子表服务器都不宜执 行任务合并而仍然频繁地从等待队列中取出待合并任务去执行任务合并,在附 图 8a至附图 8d示例的键值数据库的数据合并装置中, 将按照表 1示例的合并任 务执行策略不能执行任务合并而返回等待队列中的待合并任务返回等待队列 后, 任务冻结模块 801可以将所述待合并任务冻结, 以使所述待合并任务在设 定的冻结时间内处于不能取出所述等待队列的状态。 至于冻结时间设定的大 小, 可以根据当前系统的负载等级和 /或所述待合并任务的紧急程度等级预期 持续时间的长短而由用户设定, 本发明对此不做特別限定。 For the "saturated" level, according to the combined task execution strategy exemplified in Table 1, whether the urgency level of the task to be merged is "normal", "emergency" or "must", the merge of the tasks to be merged cannot be performed. . If, during the period of the above example, it is still frequently taken out of the waiting queue before the task is merged, the task cannot be performed and the task in the waiting queue is returned, which is obviously a waste of resources. Therefore, in order to avoid that the sub-table server is not suitable for performing task merging for a relatively long period of time and still frequently removes the task to be merged from the waiting queue to perform task merging, the data of the key value database exemplified in FIGS. 8a to 8d. In the merging device, after the merge task execution policy according to the example in Table 1 fails to perform task merging and returns to the waiting queue in the waiting queue to return to the waiting queue, the task freezing module 801 may freeze the to-be-consolidated task to make the waiting The merge task is in a state in which the wait queue cannot be fetched within the set freeze time. The size of the freeze time setting may be set by the user according to the load level of the current system and/or the expected duration of the urgency level of the task to be merged, which is not specifically limited by the present invention.
附图 8a至附图 8d示例的键值数据库的数据合并装置可以包括第一解冻模 块 901 , 如附图 9a至附图 9d所示本发明另一实施例提供的键值数据库的数据合 并装置。 第一解冻模块 901用于启动计时器对所述设定的冻结时间进行计时, 以使在所述设定的冻结时间结束后,将所述待合并任务更改为能够取出所述等 待队列的状态。 由于冻结时间是基于当前系统的负载所处等级的预期持续时间的长短而 设定, 因此,也不能排除冻结时间的设定超过当前系统的负载所处等级的预期 持续时间。 例如, 若当前系统的负载等级或 /和所述待合并任务的紧急程度等 级出现变化,从而按照表 1示例的合并任务执行策略,对之前按照表 1示例的合 并任务执行策略不能执行合并的待合并任务可以重新执行合并, 此时,如果机 械地等待设定的冻结时间结束才执行任务合并,对系统而言也是一种资源的浪 费。 因此, 附图 8a至附图 8d示例的键值数据库的数据合并装置也可以包括第二 解冻模块 1001、 第三解冻模块 1002或第四解冻模块 1003 , 如附图 10a至附图 10d 所示本发明另一实施例提供的键值数据库的数据合并装置, 其中: The data merging device of the key value database illustrated in Figures 8a to 8d may include a first defrosting module 901, such as the data merging device of the key value database provided by another embodiment of the present invention as shown in Figures 9a to 9d. The first defrosting module 901 is configured to start a timer to time the set freezing time, so that after the set freezing time ends, the to-be-combined task is changed to a state in which the waiting queue can be taken out. . Since the freeze time is set based on the length of the expected duration of the current system's load level, it is also impossible to exclude that the freeze time setting exceeds the expected duration of the current system load level. For example, if the load level of the current system or/and the urgency level of the task to be merged changes, so as according to the merge task execution policy exemplified in Table 1, the merged task execution policy according to the previous example according to Table 1 cannot be performed. The merge task can re-execute the merge. At this point, if the job merge is performed mechanically waiting for the set freeze time to end, it is also a waste of resources for the system. Therefore, the data merging device of the key value database illustrated in FIGS. 8a to 8d may also include a second defrosting module 1001, a third defrosting module 1002, or a fourth defrosting module 1003, as shown in FIG. 10a to FIG. 10d. A data merging device for a key value database provided by another embodiment, wherein:
第二解冻模块 1001 , 用于若所述当前系统的负载等级由饱和下降为高负 载、 中负载或低负载、 由高负载下降为中负载或低负载、 或者由中负载下降为 低负载, 则将所述待合并任务更改为能够取出所述等待队列的状态。  The second defrosting module 1001 is configured to: if the load level of the current system decreases from saturation to high load, medium load or low load, from high load to medium load or low load, or from medium load to low load, The task to be merged is changed to a state in which the waiting queue can be taken out.
第三解冻模块 1002,用于若所述待合并任务的紧急程度等级由非紧急上升 为普通或紧急、或者由普通上升为紧急, 则将所述待合并任务更改为能够取出 所述等待队列的状态。  The third defrosting module 1002 is configured to change the to-be-combined task to be able to take out the waiting queue if the urgency level of the task to be merged rises from non-emergency to normal or urgent, or from ordinary to urgent. status.
第四解冻模块 1003 ,用于在所述等待队列中所有待合并任务均冻结并且合 并任务执行模块处于空闲状态时,若所述待合并任务的紧急程度等级比其他待 合并任务的紧急程度等级高,则将所述待合并任务更改为能够取出所述等待队 列的状态。  The fourth defrosting module 1003 is configured to: when all the tasks to be merged are frozen in the waiting queue and the merging task execution module is in an idle state, if the urgency level of the task to be merged is higher than the urgency level of other tasks to be merged And changing the to-be-combined task to a state in which the waiting queue can be taken out.
需要说明的是, 上述装置各模块 /单元之间的信息交互、 执行过程等内容, 由于与本发明方法实施例基于同一构思,其带来的技术效果与本发明方法实施 例相同, 具体内容可参见本发明方法实施例中的叙述, 此处不再赘述。  It should be noted that the information interaction, the execution process, and the like between the modules/units of the foregoing device are the same as the embodiment of the method of the present invention. Reference is made to the description in the method embodiment of the present invention, and details are not described herein again.
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步 骤是可以通过程序来指令相关的硬件来完成,比如以下各种方法的一种或多种 或全部:  One of ordinary skill in the art will appreciate that all or a portion of the various methods of the above-described embodiments can be performed by a program to instruct the associated hardware, such as one or more or all of the following various methods:
获取当前系统的负载等级和待合并任务的紧急程度等级;  Obtain the current system's load level and the urgency level of the task to be merged;
根据所述当前系统的负载等级和待合并任务的紧急程度等级,匹配合并任 略为当前系统的负载等级越高, 允许执行紧急程度等级越高的待合并任务, 在 当前系统的负载等级为饱和时, 不允许执行任何待合并任务; According to the load level of the current system and the urgency level of the task to be merged, matching and merging The higher the load level of the current system, the higher the urgency level is allowed to perform the tasks to be merged. When the load level of the current system is saturated, no tasks to be merged are allowed.
若确定执行所述待合并任务,则将所述待合并任务发送至合并任务执行模 块执行合并任务。  If it is determined that the to-be-combined task is executed, the to-be-combined task is sent to the merged task execution module to perform a merge task.
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步 骤是可以通过程序来指令相关的硬件来完成,该程序可以存储于一计算机可读 存储介质中, 存储介质可以包括: 只读存储器( ROM, Read Only Memory ) 、 随机存取存储器(RAM, Random Access Memory ) 、 磁盘或光盘等。  A person skilled in the art may understand that all or part of the various steps of the foregoing embodiments may be completed by a program instructing related hardware. The program may be stored in a computer readable storage medium, and the storage medium may include: Read Only Memory (ROM), Random Access Memory (RAM), disk or optical disk.
以上对本发明实施例提供的键值数据库的数据合并方法和装置进行了详 实施例的说明只是用于帮助理解本发明的方法及其核心思想; 同时,对于本领 域的一般技术人员,依据本发明的思想,在具体实施方式及应用范围上均会有 改变之处, 综上所述, 本说明书内容不应理解为对本发明的限制。  The foregoing detailed description of the data merging method and apparatus for the key value database provided by the embodiment of the present invention is only for helping to understand the method of the present invention and its core idea; meanwhile, for those skilled in the art, according to the present invention The scope of the present invention is not limited by the scope of the present invention.

Claims

权 利 要 求 Rights request
1、 一种键值数据库的数据合并方法, 其特征在于, 所述方法包括: 获取当前系统的负载等级和待合并任务的紧急程度等级; 1. A data merging method for a key-value database, characterized in that the method includes: obtaining the load level of the current system and the urgency level of the tasks to be merged;
根据所述当前系统的负载等级和待合并任务的紧急程度等级,匹配合并任 略为当前系统的负载等级越高, 允许执行紧急程度等级越高的待合并任务, 在 当前系统的负载等级为饱和时, 不允许执行任何待合并任务; According to the load level of the current system and the urgency level of the tasks to be merged, the matching merging option is that the higher the load level of the current system, the higher the urgency level of the tasks to be merged is allowed to be executed, when the load level of the current system is saturated. , no tasks to be merged are allowed to be executed;
若确定允许执行所述待合并任务,则将所述待合并任务发送至合并任务执 行模块执行任务合并。 If it is determined that the task to be merged is allowed to be executed, the task to be merged is sent to the merge task execution module to perform task merge.
2、 如权利要求 1所述的方法, 其特征在于, 所述获取当前系统的负载等级 包括: 2. The method of claim 1, wherein obtaining the load level of the current system includes:
将当前系统的资源的使用情况按照权重进行加权求和,根据所述加权求和 所得的值对所述当前系统的负载赋予相应的等级,所述系统的资源的使用情况 包括 CPU的使用情况、 内存使用情况、 网络使用情况和磁盘使用情况的任意组 合, 所述当前系统的负载等级按照低到高依次包括低负载、 中负载、 高负载和 饱和。 The resource usage of the current system is weighted and summed according to the weight, and the load of the current system is assigned a corresponding level according to the value obtained by the weighted sum. The resource usage of the system includes the usage of CPU, Any combination of memory usage, network usage and disk usage, the load level of the current system includes low load, medium load, high load and saturation in order from low to high.
3、 如权利要求 1所述的方法, 其特征在于, 所述获取待合并任务的紧急程 度等级包括: 3. The method of claim 1, wherein obtaining the urgency level of tasks to be merged includes:
将影响所述待合并任务的紧急程度的各因素按照权重进行线性加权求和, 根据所述线性加权求和所得的值对所述待合并任务的紧急程度赋予相应的等 级, 所述影响所述待合并任务的紧急程度的各因素包括子表中的文件大小、子 表中文件个数和子表中最老数据的超期时间的任意组合,所述待合并任务的紧 急程度等级包括非紧急、 普通和紧急。 The factors that affect the urgency of the tasks to be merged are linearly weighted and summed according to the weights, and the urgency of the tasks to be merged is assigned a corresponding level according to the value obtained by the linear weighted sum. The factors of the urgency of the tasks to be merged include any combination of the file size in the sub-table, the number of files in the sub-table and the expiration time of the oldest data in the sub-table. The urgency level of the tasks to be merged includes non-urgent, ordinary and urgent.
4、 如权利要求 1所述的方法, 其特征在于, 若当前影响所述待合并任务的 紧急程度的各因素中的因素¾的影响值已达到或接近所述因素¾的最大值,则 所述获取待合并任务的紧急程度等级包括: 4. The method according to claim 1, characterized in that, if the influence value of factor ¾ among the factors that currently affect the urgency of the task to be merged has reached or is close to the maximum value of the factor ¾, then the The urgency levels of tasks to be merged include:
将影响所述待合并任务的紧急程度的各因素按照权重进行非线性加权求 和,根据所述非线性加权求和所得的值对所述待合并任务的紧急程度赋予相应 的等级, 所述影响所述待合并任务的紧急程度的各因素包括子表中的文件大 小、子表中文件个数和子表中最老数据的超期时间的任意组合, 所述待合并任 务的紧急程度等级包括非紧急、 普通和紧急。 The factors that affect the urgency of the tasks to be merged are subjected to a nonlinear weighted sum according to the weights, and the urgency of the tasks to be merged is assigned a corresponding value based on the value obtained by the nonlinear weighted sum. The factors that affect the urgency of the tasks to be merged include any combination of the file size in the sub-table, the number of files in the sub-table and the expiration time of the oldest data in the sub-table, the tasks to be merged are Urgency levels include non-urgent, normal and urgent.
5、 如权利要求 4所述的方法, 其特征在于, 所述将影响所述待合并任务的 紧急程度的各因素按照权重进行非线性加权求和包括: 5. The method of claim 4, wherein the nonlinear weighted summation of factors that affect the urgency of the tasks to be merged according to weights includes:
将影响所述待合并任务的紧急程度的因素的权重 ^放大至 , 所述 为权重放大函数; Amplify the weight ^ of the factors that affect the urgency of the tasks to be merged to , where ^ is the weight amplification function;
以所述 作为影响所述待合并任务的紧急程度的因素新的权重, 将 影响所述待合并任务的紧急程度的因素按照所述新的权重进行非线性加权求 和,根据所述非线性加权求和所得的值对所述待合并任务的紧急程度赋予相应 的等级, 所述待合并任务的紧急程度等级包括非紧急、 普通和紧急。 Using the new weights as factors that affect the urgency of the tasks to be merged, perform a nonlinear weighted sum of the factors that affect the urgency of the tasks to be merged according to the new weights, and based on the nonlinear weighting The value obtained by the summation assigns a corresponding level to the urgency of the tasks to be merged, and the urgency levels of the tasks to be merged include non-urgent, normal and urgent.
6、 如权利要求 2至 5任意一项所述的方法, 其特征在于, 所述根据所述当 前系统的负载等级和待合并任务的紧急程度等级,匹配合并任务执行策略以确 定是否允许执行所述待合并任务包括: 6. The method according to any one of claims 2 to 5, characterized in that, according to the load level of the current system and the urgency level of the tasks to be merged, the merged task execution policy is matched to determine whether to allow execution of the tasks to be merged. Tasks to be merged include:
若匹配到所述待合并任务的紧急程度等级为紧急,在所述当前系统的负载 等级为低负载、 中负载或高负载时, 则确定允许执行所述待合并任务; If the matching urgency level of the task to be merged is urgent, and when the load level of the current system is low load, medium load or high load, it is determined that the task to be merged is allowed to be executed;
若匹配到所述待合并任务的紧急程度等级为非紧急,在所述当前系统的负 载等级为中负载或中负载以上时, 则确定不允许执行所述待合并任务; If the urgency level of the task to be merged is matched to non-emergency, and the load level of the current system is medium load or above, it is determined that the task to be merged is not allowed to be executed;
若匹配到所述待合并任务的紧急程度等级为普通,在所述当前系统的负载 等级为高负载或高负载以上时, 则确定不允许执行所述待合并任务。 If the matching urgency level of the task to be merged is normal and the load level of the current system is high load or above, it is determined that the task to be merged is not allowed to be executed.
7、 如权利要求 2至 5任意一项所述的方法, 其特征在于, 所述将所述待合 并任务返回等待队列后进一步包括: 7. The method according to any one of claims 2 to 5, wherein returning the tasks to be merged to the waiting queue further includes:
将所述待合并任务冻结,以使所述待合并任务在设定的冻结时间内处于不 能取出所述等待队列的状态。 The tasks to be merged are frozen so that the tasks to be merged are in a state where they cannot be taken out of the waiting queue within the set freezing time.
8、 如权利要求 7所述的方法, 其特征在于, 所述将所述待合并任务冻结后 进一步包括: 8. The method of claim 7, wherein the freezing of the tasks to be merged further includes:
启动计时器对所述设定的冻结时间进行计时,以使在所述设定的冻结时间 结束后, 将所述待合并任务更改为能够取出所述等待队列的状态。 A timer is started to count the set freezing time, so that after the set freezing time ends, the tasks to be merged are changed to a state in which the waiting queue can be taken out.
9、 如权利要求 7所述的方法, 其特征在于, 所述将所述待合并任务冻结后 进一步包括: 9. The method of claim 7, wherein the freezing of the tasks to be merged further includes:
若所述当前系统的负载等级由饱和下降为高负载、 中负载或低负载、 由高 负载下降为中负载或低负载、或者由中负载下降为低负载, 则将所述待合并任 务更改为能够取出所述等待队列的状态; 或者 If the load level of the current system drops from saturation to high load, medium load or low load, from high load to medium load or low load, or from medium load to low load, then the tasks to be merged are changed to Be able to retrieve the status of the waiting queue; or
若所述待合并任务的紧急程度等级由非紧急上升为普通或紧急,或者由普 通上升为紧急, 则将所述待合并任务更改为能够取出所述等待队列的状态; 或 者 If the urgency level of the tasks to be merged is raised from non-urgent to normal or urgent, or from normal to urgent, then the tasks to be merged are changed to a state where the waiting queue can be taken out; or
在所述等待队列中所有待合并任务均冻结并且合并任务执行模块处于空 闲状态时,若所述待合并任务的紧急程度等级比其他待合并任务的紧急程度等 级高, 则将所述待合并任务更改为能够取出所述等待队列的状态。 When all tasks to be merged in the waiting queue are frozen and the merge task execution module is in an idle state, if the urgency level of the task to be merged is higher than the urgency level of other tasks to be merged, the task to be merged will be Change to a state where the waiting queue can be removed.
10、 一种键值数据库的数据合并装置, 其特征在于, 所述装置包括: 获取模块, 用于获取当前系统的负载等级和待合并任务的紧急程度等级; 确定模块,用于根据所述当前系统的负载等级和待合并任务的紧急程度等 合并任务执行策略为当前系统的负载等级越高,允许执行紧急程度等级越高的 待合并任务, 在当前系统的负载等级为饱和时, 不允许执行任何待合并任务; 处理模块, 用于若所述确定模块确定执行所述待合并任务, 则将所述待合 并任务发送至合并任务执行模块执行任务合并。 10. A data merging device for a key-value database, characterized in that the device includes: an acquisition module, used to acquire the load level of the current system and the urgency level of the tasks to be merged; a determination module, used to obtain the load level of the current system and the urgency level of the tasks to be merged; a determination module, used to determine according to the current The load level of the system and the urgency of the tasks to be merged are determined by the merge task execution policy. The higher the load level of the current system, the higher the urgency level of the tasks to be merged is allowed to be executed. When the load level of the current system is saturated, execution is not allowed. Any task to be merged; a processing module, configured to send the task to be merged to the merge task execution module to perform task merge if the determination module determines to execute the task to be merged.
11、 如权利要求 10所述的装置, 其特征在于, 所述获取模块包括: 第一获取单元, 用于将当前系统的资源的使用情况按照权重进行加权求 和,根据所述加权求和所得的值对所述当前系统的负载赋予相应的等级, 所述 系统的资源的使用情况包括 CPU的使用情况、 内存使用情况、 网络使用情况和 磁盘使用情况的任意组合,所述当前系统的负载等级按照低到高依次包括低负 载、 中负载、 高负载和饱和。 11. The device according to claim 10, wherein the acquisition module includes: a first acquisition unit, configured to perform a weighted sum of the current system resource usage according to weights, and obtain the result according to the weighted sum. The value of assigns a corresponding level to the load of the current system. The resource usage of the system includes any combination of CPU usage, memory usage, network usage and disk usage. The load level of the current system The order from low to high includes low load, medium load, high load and saturation.
12、 如权利要求 10所述的装置, 其特征在于, 所述获取模块包括: 第二获取单元,用于将影响所述待合并任务的紧急程度的因素按照权重进 行线性加权求和,根据所述线性加权求和所得的值对所述待合并任务的紧急程 度赋予相应的等级,所述影响所述待合并任务的紧急程度的各因素包括子表中 的文件大小、子表中文件个数和子表中最老数据的超期时间的任意组合, 所述 待合并任务的紧急程度等级包括普通、 紧急和必须。 12. The device according to claim 10, wherein the acquisition module includes: a second acquisition unit, configured to linearly weight and sum the factors that affect the urgency of the tasks to be merged according to the weights. The value obtained by the linear weighted summation has an impact on the emergency process of the tasks to be merged. The degree is assigned a corresponding level, and the factors that affect the urgency of the task to be merged include any combination of the file size in the sub-table, the number of files in the sub-table, and the expiration time of the oldest data in the sub-table. The urgency levels of merged tasks include normal, urgent and necessary.
13、 如权利要求 10所述的装置, 其特征在于, 若当前影响所述待合并任务 的紧急程度的各因素中的因素¾的影响值已达到或接近所述因素¾的最大值, 则所述获取模块包括: 13. The device according to claim 10, wherein if the influence value of factor ¾ among the factors that currently affect the urgency of the task to be merged has reached or is close to the maximum value of the factor ¾, then the The above acquisition modules include:
第三获取单元,用于将影响所述待合并任务的紧急程度的各因素按照权重 进行非线性加权求和,根据所述加权求和所得的值对所述待合并任务的紧急程 度赋予相应的等级,所述影响所述待合并任务的紧急程度的各因素包括子表中 的文件大小、子表中文件个数和子表中最老数据的超期时间的任意组合, 所述 待合并任务的紧急程度等级包括非紧急、 普通和紧急。 The third acquisition unit is configured to perform a nonlinear weighted sum of the factors that affect the urgency of the tasks to be merged according to weights, and assign a corresponding value to the urgency of the tasks to be merged based on the value obtained by the weighted sum. Level, the factors that affect the urgency of the tasks to be merged include any combination of the file size in the sub-table, the number of files in the sub-table and the expiration time of the oldest data in the sub-table, the urgency of the tasks to be merged Levels include non-urgent, normal and urgent.
14、 如权利要求 13所述的装置, 其特征在于, 所述第三获取单元包括: 放大单元,用于将影响所述待合并任务的紧急程度的因素的权重 ^放大至 f {Ak )wk , 所述 为权重放大函数; 14. The device according to claim 13, wherein the third acquisition unit includes: an amplification unit, configured to amplify the weights of factors affecting the urgency of the tasks to be merged to f {A k ) w k , said is the weight amplification function;
求和单元, 用于以所述 作为影响所述待合并任务的紧急程度的因 素新的权重,将影响所述待合并任务的紧急程度的因素按照所述新的权重进行 非线性加权求和,根据所述非线性加权求和所得的值对所述待合并任务的紧急 程度赋予相应的等级,所述待合并任务的紧急程度等级包括普通、紧急和必须。 The summation unit is configured to use the new weights as the factors that affect the urgency of the tasks to be merged, and perform a nonlinear weighted sum of the factors that affect the urgency of the tasks to be merged according to the new weights, The urgency of the tasks to be merged is assigned a corresponding level according to the value obtained by the nonlinear weighted sum. The urgency levels of the tasks to be merged include ordinary, urgent and necessary.
15、 如权利要求 11至 14任意一项所述的装置, 其特征在于, 所述确定模块 包括: 15. The device according to any one of claims 11 to 14, characterized in that the determining module includes:
第一确定单元, 用于若匹配到所述待合并任务的紧急程度等级为紧急,在 所述当前系统的负载等级为低负载、 中负载或高负载时, 则确定允许执行所述 待合并任务; A first determination unit configured to determine whether the task to be merged is allowed to be executed if the emergency level of the task to be merged is urgent and the load level of the current system is low load, medium load or high load. ;
第二确定单元, 用于若匹配到所述待合并任务的紧急程度等级为非紧急, 在所述当前系统的负载等级为中负载或中负载以上时,则确定不允许执行所述 待合并任务; The second determination unit is configured to determine that if the emergency level of the task to be merged is non-urgent and the load level of the current system is medium load or above, it is determined that the task to be merged is not allowed to be executed. ;
第三确定单元, 用于若匹配到所述待合并任务的紧急程度等级为普通,在 所述当前系统的负载等级为高负载或高负载以上时,则确定不允许执行所述待 合并任务的合并。 The third determination unit is configured to determine that if the emergency level of the tasks to be merged is normal and the load level of the current system is high load or above, it is determined that the execution of the tasks to be merged is not allowed. Merge of merge tasks.
16、 如权利要求 11至 14任意一项所述的装置, 其特征在于, 所述装置进一 步包括: 16. The device according to any one of claims 11 to 14, characterized in that the device further includes:
任务冻结模块, 用于将所述待合并任务冻结, 以使所述待合并任务在设定 的冻结时间内处于不能取出所述等待队列的状态。 A task freezing module is used to freeze the tasks to be merged, so that the tasks to be merged are in a state where they cannot be taken out of the waiting queue within the set freezing time.
17、 如权利要求 16所述的装置, 其特征在于, 所述装置进一步包括: 第一解冻模块, 用于启动计时器对所述设定的冻结时间进行计时, 以使在 所述设定的冻结时间结束后,将所述待合并任务更改为能够取出所述等待队列 的状态。 17. The device according to claim 16, wherein the device further includes: a first thawing module, configured to start a timer to time the set freezing time, so that the set freezing time is After the freezing time is over, the tasks to be merged are changed to a state where they can be taken out of the waiting queue.
18、 如权利要求 16所述的装置, 其特征在于, 所述装置进一步包括第二解 冻模块、 第三解冻模块或第四解冻模块; 18. The device according to claim 16, wherein the device further includes a second thawing module, a third thawing module or a fourth thawing module;
所述第二解冻模块, 用于若所述当前系统的负载等级由饱和下降为高负 载、 中负载或低负载、 由高负载下降为中负载或低负载、 或者由中负载下降为 低负载, 则将所述待合并任务更改为能够取出所述等待队列的状态; The second unfreezing module is used if the load level of the current system drops from saturation to high load, medium load or low load, from high load to medium load or low load, or from medium load to low load, Then change the task to be merged to a state where the waiting queue can be taken out;
所述第三解冻模块,用于若所述待合并任务的紧急程度等级由非紧急上升 为普通或紧急, 或者由普通上升为紧急, 则将所述待合并任务更改为能够取出 所述等待队列的状态; The third unfreezing module is used to change the task to be merged to be able to take out the waiting queue if the urgency level of the task to be merged is raised from non-urgent to normal or urgent, or from normal to urgent. status;
所述第四解冻模块,用于在所述等待队列中所有待合并任务均冻结并且合 并任务执行模块处于空闲状态时,若所述待合并任务的紧急程度等级比其他待 合并任务的紧急程度等级高,则将所述待合并任务更改为能够取出所述等待队 列的状态。 The fourth unfreezing module is used to, when all tasks to be merged in the waiting queue are frozen and the merge task execution module is in an idle state, if the urgency level of the task to be merged is higher than the urgency level of other tasks to be merged If it is high, the task to be merged will be changed to a state where the waiting queue can be taken out.
PCT/CN2012/077314 2012-06-21 2012-06-21 Key-value database data merging method and device WO2013189063A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2012/077314 WO2013189063A1 (en) 2012-06-21 2012-06-21 Key-value database data merging method and device
CN201280000718.XA CN103649948A (en) 2012-06-21 2012-06-21 Key-value database data merging method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2012/077314 WO2013189063A1 (en) 2012-06-21 2012-06-21 Key-value database data merging method and device

Publications (1)

Publication Number Publication Date
WO2013189063A1 true WO2013189063A1 (en) 2013-12-27

Family

ID=49768044

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2012/077314 WO2013189063A1 (en) 2012-06-21 2012-06-21 Key-value database data merging method and device

Country Status (2)

Country Link
CN (1) CN103649948A (en)
WO (1) WO2013189063A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105489216A (en) * 2016-01-19 2016-04-13 百度在线网络技术(北京)有限公司 Voice synthesis system optimization method and device

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101894163A (en) * 2010-07-23 2010-11-24 中兴通讯股份有限公司 Database operating and scheduling method and device for performance data acquisition system

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101894163A (en) * 2010-07-23 2010-11-24 中兴通讯股份有限公司 Database operating and scheduling method and device for performance data acquisition system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
FEI, JIANGTAO ET AL.: "Technology Summarization of Mass-data Management System Based on Table Structure", COMPUTER AND MODERNIZATION, no. 1, January 2010 (2010-01-01), pages 166 - 169 *
WANG, YINGDONG ET AL.: "The Research for Load Balancing Technology of Bigtable System", NETWORK AND COMPUTER SECURITY, no. 2, February 2009 (2009-02-01), pages 41 - 43 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105489216A (en) * 2016-01-19 2016-04-13 百度在线网络技术(北京)有限公司 Voice synthesis system optimization method and device
US10242660B2 (en) 2016-01-19 2019-03-26 Baidu Online Network Technology (Beijing) Co., Ltd. Method and device for optimizing speech synthesis system

Also Published As

Publication number Publication date
CN103649948A (en) 2014-03-19

Similar Documents

Publication Publication Date Title
US11593404B2 (en) Multi-cluster warehouse
KR101721892B1 (en) Managing queries
US9286116B2 (en) Multiple core real-time task execution
JP5733680B2 (en) Command execution device, command execution system, command execution method, and command execution program
JP5949937B2 (en) Transaction system
CN110781244B (en) Method and device for controlling concurrent operation of database
US20170308567A1 (en) Resource Coordination Method, Apparatus, and System for Database Cluster
US10142195B1 (en) Partitioned performance tracking core resource consumption independently
US10148531B1 (en) Partitioned performance: adaptive predicted impact
WO2022062184A1 (en) High-concurrency query method, intelligent terminal and storage medium
US20170357681A1 (en) Dynamic in-flight database request throttling
US10033620B1 (en) Partitioned performance adaptive policies and leases
WO2013189063A1 (en) Key-value database data merging method and device
WO2023066070A1 (en) Sql statement control method and apparatus
JP6046523B2 (en) In-memory distributed database, data distribution method and program
US20080282247A1 (en) Method and Server for Synchronizing a Plurality of Clients Accessing a Database
Luo et al. On performance stability in LSM-based storage systems (extended version)
JP6459654B2 (en) Event-driven system, information processing apparatus, event-driven program, and event-driven method
US11271992B2 (en) Lazy lock queue reduction for cluster group changes
US9503353B1 (en) Dynamic cross protocol tuner
JP5747770B2 (en) Data storage control device, data storage control program, and data storage control method
US20150212749A1 (en) Methods for managing read access of objects in storage media and devices thereof
US10148588B1 (en) Partitioned performance: using resource account aggregates to throttle at the granular level
Yeh et al. Improving the program performance through prioritized memory management and disk operation
Xia et al. An optimized load balance based on data popularity on HBASE

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12879170

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12879170

Country of ref document: EP

Kind code of ref document: A1