WO2013189063A1 - Procédé et dispositif de fusionnement de données d'une base de données de valeurs de clés - Google Patents

Procédé et dispositif de fusionnement de données d'une base de données de valeurs de clés Download PDF

Info

Publication number
WO2013189063A1
WO2013189063A1 PCT/CN2012/077314 CN2012077314W WO2013189063A1 WO 2013189063 A1 WO2013189063 A1 WO 2013189063A1 CN 2012077314 W CN2012077314 W CN 2012077314W WO 2013189063 A1 WO2013189063 A1 WO 2013189063A1
Authority
WO
WIPO (PCT)
Prior art keywords
merged
task
tasks
urgency
load
Prior art date
Application number
PCT/CN2012/077314
Other languages
English (en)
Chinese (zh)
Inventor
周帅锋
智伟
赵智峰
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN201280000718.XA priority Critical patent/CN103649948A/zh
Priority to PCT/CN2012/077314 priority patent/WO2013189063A1/fr
Publication of WO2013189063A1 publication Critical patent/WO2013189063A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application

Definitions

  • the present invention relates to the field of data processing, and in particular, to a data merging method and apparatus for a key value database. Background technique
  • the Key-Value database is a database management system based on key-value data storage.
  • Key-value data storage technology stores data in a key-value pair data model that provides persistence mechanisms and data synchronization.
  • data is stored sequentially in terms of keys (Key) and written to disk in chasing mode.
  • BigTable and HBase are the modes. In this mode, files written to the file system are not modified once they are written. Thus, data written at different times will form multiple storage files in the file system.
  • the key-value database will start the merge operation and merge multiple files after the storage file reaches a certain size or for a period of time. At the same time, some data cleaning work is done.
  • sub-tables are the basic unit for performing data merge tasks, and sub-tables are usually managed by multiple sub-table servers.
  • the data table is divided into a number of sub-tables according to the scope of the key. Each sub-table stores and manages all data whose key values fall within its range.
  • the data merging method in a key value database is: using a sub-table as a basic unit of scheduling execution, and when the number of files in the sub-table reaches or exceeds a predefined upper limit, the merging task is triggered, and the sub-task is triggered.
  • the table is placed in the data merge execution queue, from which the data merge thread obtains the child table, and the merge is performed, as shown in FIG.
  • Embodiments of the present invention provide a data merging method and apparatus for a key value database, and control the data merging task to avoid a large amount of data merging tasks while burdening the system.
  • An embodiment of the present invention provides a data merging method for a key value database, where the method includes: acquiring The load level of the current system and the urgency level of the task to be merged;
  • the task to be merged is sent to the merge task execution module to perform task merge.
  • An embodiment of the present invention provides a data merging device for a key value database, where the device includes: an acquiring module, configured to acquire a load level of a current system and an urgency level of a task to be merged;
  • a determining module configured to merge the task execution policy according to the load level of the current system and the urgency of the task to be merged, etc., the higher the load level of the current system, the higher the urgency level is allowed to perform the task to be merged, in the current system When the load level is saturated, the task to be merged is not allowed to be executed; and the processing module is configured to send the to-be-consolidated task to the merge task execution module to perform task combination if the determining module determines to perform the to-be-combined task.
  • the merging is allowed according to the load level of the current system and the urgency level of the task to be merged according to the load level of the current system and the merging task execution strategy, that is, the urgency is allowed according to the system resource load level.
  • the method provided by the embodiment of the present invention can control the data merging task according to the system resource occupation, so that the data merging task is automatically delayed when the system load is high, thereby reducing the impact of the execution task merging on the access service and reducing the system burden.
  • FIG. 1 is a schematic diagram of a data merging method in a key value database provided by the prior art
  • FIG. 2 is a schematic flowchart of a data merging method of a key value database according to an embodiment of the present invention
  • 3 is a schematic structural diagram of a data merging device of a key value database according to an embodiment of the present invention
  • FIG. 4 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 6 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 6b is a data merging device for a key value database according to another embodiment of the present invention
  • FIG. 7 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 7b is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 7 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8 is a structural diagram of a data merging device according to another embodiment of the present invention
  • a schematic diagram of a data merging device of a database FIG. 8b is a key value provided by another embodiment of the present invention
  • FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 8
  • FIG. 9 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 9b is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 9 is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention
  • FIG. 10 is a schematic diagram of a data merging database provided by another embodiment of the present invention; Schematic diagram of the data merging device;
  • FIG. 10b is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention.
  • FIG. 10c is a schematic structural diagram of a data merging device of a key value database according to another embodiment of the present invention.
  • Figure 10d is a schematic diagram showing the structure of a data merging device of a key value database according to another embodiment of the present invention.
  • Embodiments of the present invention provide a data merging method and apparatus for a key value database to schedule a data merging task according to system resource occupancy.
  • the technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without departing from the inventive scope are the scope of the present invention.
  • the executor may be a sub-table server in a key value database, and mainly includes step S201, step S202, and step S203:
  • the merge task of the key value database is related to the load of the current system, for example, when the system is busy and/or the system load is relatively high, the unconstrained execution of the key database database merge task will increase the system burden and affect the data access service. Performance.
  • whether or not the merge task of the key-value database is executed is also related to the urgency of the task to be merged. Therefore, in the embodiment of the present invention, the load level of the current system and the urgency level of the task to be merged may be first acquired, where the load level of the current system includes but is not limited to low load, medium load, high load, and saturation.
  • the urgency levels of the tasks to be combined include, but are not limited to, non-emergency, normal, and emergency.
  • the usage of the resources of the current system may be weighted and summed according to the weight, and the load of the current system is given a corresponding level according to the value obtained by the weighted summation.
  • a central processing unit CPU
  • a memory e.g., a RAM
  • a network e.g., a graphics processing unit
  • a disk e.g., a hard disk
  • the weights of the resources may be weighted.
  • the weighted sum is equivalent to the score of the current system load. For example, if the current system's load score is represented by L, then: where ⁇ and ⁇ respectively represent the CPU score weight, the memory score weight, and the network score weight.
  • disk scoring weights, and u cpu u m t ⁇ and respectively indicate CPU usage, memory usage, network usage, and disk usage.
  • the load of the system can be divided into low load, medium load, high load and saturation according to the level from low to high
  • the current system load can be assigned to the corresponding level according to the system load score L from low to high. , for example, if the load on the system When the score L is 100, the load of the system can be scored in the range of 0 ⁇ 25, 25 - 50, 50 ⁇ 75 and 75 ⁇ 100, and the current system load level is sequentially assigned to the low load, medium load, High load and saturation.
  • the factors affecting the urgency of the task to be merged may be linearly weighted and summed according to the weight, and the value to be merged according to the value obtained by the linear weighted summation
  • the urgency of the task corresponds to the level.
  • factors affecting the urgency of the task to be merged include, but are not limited to, the file size in the sub-table, the number of files in the sub-table, and the overdue time of the oldest data in the sub-table. Since there are quantitative differences in different influencing factors, the urgency score of the tasks to be combined can be calculated by linear weighted summation. In this way, for each factor affecting the urgency of the task to be combined, the percentage is calculated based on the maximum value of each factor and weighted and summed, as shown below, if the urgency score of the task to be combined is used
  • Score ⁇ X l , X 2 ,...,X k ) j w k ⁇ X k I MAX Xk ) ⁇ lQQ%
  • ⁇ represents the first factor affecting the urgency of the task to be merged
  • ⁇ Indicates the weight of the first factor affecting the urgency of the task to be merged
  • MX indicates the maximum value that the first factor affecting the urgency of the task to be merged can reach or set.
  • the urgency level of the task to be merged can be divided into non-emergency, normal and emergency, etc., it can be scored according to the urgency of the task to be merged ⁇ ( ⁇ ,..., ⁇ )
  • the urgency of the task to be merged is assigned to the corresponding level in turn, for example, if the urgency score of the task to be combined is ⁇ , ⁇ , ..., ⁇ ) is 100, the urgency degree of the task to be combined can be scored. 1 3 ⁇ 4 ⁇ ( 1 , 2 ,..., ⁇ ) In the interval of 0 ⁇ 30, 30 ⁇ 60 and 60 ⁇ 100, the urgency level of the task to be merged is sequentially assigned to non-emergency, normal and emergency.
  • the urgency of the combined task is scored, considering the general state of each factor in the common scenario. However, in some scenarios, the scores of other factors affecting the urgency of the task to be merged may still be If the factors affecting the urgency of the task to be merged are close to or reach the maximum value, the overall impact of the urgency of the execution combination of the tasks to be merged begins to increase non-linearly. Due to the urgency of the formula (2) to treat the combined task Is a linear weighted summation. When the factor affecting the urgency of the task to be merged is close to or reaches the maximum value, if the urgency of the combined task is still scored according to formula (2), the scored result cannot be faithfully reflected. The urgency of the merger to be merged is obviously unreasonable.
  • the embodiment of the invention provides a scoring method for a single item affecting the extreme value amplification weighted summation, so as to achieve a non-linear enlargement of the urgency degree of the combined task when the urgency affecting the task to be combined individually reaches or approaches the maximum value. That is, if the influence value of one of the factors affecting the urgency of the task to be merged has reached or is close to the maximum value of the factor, the factors affecting the urgency of the task to be merged are weighted according to the weight.
  • the nonlinear weighted summation assigns a corresponding level to the urgency of the task to be merged according to the value obtained by the nonlinear weighted summation.
  • the nonlinear weighted summation of each factor affecting the urgency of the task to be merged according to the weight includes: enlarging the weight of the factor affecting the urgency of the task to be merged to a new weight for the urgency of the combined task, a factor that affects the urgency of the task to be merged, a nonlinear weighted summation according to the new weight, and a value pair obtained by the nonlinear weighted summation
  • the urgency level of the task to be merged is the urgency score of the task to be merged ⁇ / ⁇ : ⁇ ,: ⁇ ,...,: ⁇ ;! Expressed as:
  • Equation (4) MAX Xt /(MAX Xt -X k +AX k ) Equation (4)
  • MAX Xt represents the maximum value of the influence factor of the first urgency affecting the task to be combined
  • ⁇ ⁇ is a minimum value of the first factor affecting the urgency of the task to be merged, and is used to adjust equation (4) such that the denominator of equation (4) is not zero.
  • the load level of the current system obtained by the above-mentioned score calculated according to the formula (1), and the urgency level of the task to be combined obtained by the score calculated according to the formula (2) or the formula (3) are only For example, it is not limited to the formula (1), (2) or (3) above. Style.
  • the principle of determining a set of tasks for performing task consolidation includes: The higher the load level of the current system, the higher the urgency level is allowed to perform the task to be merged, that is, the task to be merged is allowed to be higher.
  • the merge operation is performed. When the load level of the current system is saturated, no tasks to be merged are allowed to be executed, that is, the merge of any tasks to be merged is not allowed.
  • the execution strategy of the merged task can also be interpreted as: When the urgency level of the task to be merged is determined, the higher the load level of the current system, the lower the possibility that the task to be merged is allowed to perform the merge, and vice versa, the current system The lower the load level, the higher the probability that the task to be merged is allowed to perform the merge; or the higher the urgency level of the task to be merged when the load level of the current system is determined (ie, the degree to which the task to be merged needs to perform the merge) The more urgent, the higher the probability that the task to be merged is allowed to perform the merge, and the lower the level of urgency of the task to be merged (that is, the more relaxed the degree of the merge to be merged), the task to be merged is not allowed.
  • the urgency level of the tasks to be merged is merged; when the urgency level of the task to be merged is the highest, for example, when the urgency level of the task to be merged is urgent, unless the load level of the current system is the highest, for example, the load of the current system The level is saturated, otherwise, the merging is allowed for the task to be merged with the urgency level being urgent.
  • the current system's load levels include, but are not limited to, low load, medium load, high load, and saturation
  • the urgency levels of the tasks to be combined include, but are not limited to, non-emergency, normal, and emergency.
  • the merge task execution policy provided by the embodiment of the present invention may be as shown in Table 1 below:
  • the "load level” is the cartridge of the "current system load level” in the foregoing embodiment, including “low load”, “medium load”, “high load” and “saturated”, “urgency level”.
  • " is the nickname of the urgency level of the task to be merged in the foregoing embodiment, including “non-emergency", “normal”, and “emergency”.
  • the item at the intersection of "load level” and “urgency level” indicates whether to perform the combination of the tasks to be merged. For example, when the "load level” is “low load” and the “urgency level” is “non-emergency", Perform a merge of the tasks to be merged.
  • the general principle of the combined task execution strategy reflected in Table 1 above is that for a task to be merged with a urgency level of non-emergency, when the load level of the system is above the medium load or the medium load, the merged service is not allowed to be merged.
  • the task to be merged is returned to the queue and waits; for the task to be merged with the urgency level is normal, when the load level of the system is high load or high load, the merged service is not allowed to be merged, but
  • the to-be-consolidated task returns to the queue to wait; for the to-be-consolidated task whose urgency level is urgent, as long as the load level of the system is not saturated, that is, when the current system load level is low load, medium load or high load, the to-be-committed is allowed.
  • Business execution merges.
  • matching the merge task execution policy to determine whether to allow the task to be merged includes: if the urgency level matching the task to be merged is urgent, When the load level of the current system is low load, medium load or high load, it is determined that the task to be merged is allowed to be executed, that is, it is determined that the merge is allowed to be performed on the task to be merged; if the task is to be merged to the task to be merged If the level of the current system is non-emergency, when the load level of the current system is above the medium load or the medium load, it is determined that the task to be merged is not allowed to be executed, that is, it is determined that the merged task is not allowed to be merged; The urgency level assigned to the task to be merged is normal. When the load level of the current system is high load or high load, it is determined that the task to be merged is not allowed to be executed, that is, the waiting for the waiting is not allowed.
  • the merge task execution policy of the example in Table 1 For the merge task execution policy of the example in Table 1, if the load level of the current system acquired in step S201 is "low load”, the merge task execution policy of the example of Table 1 is matched, regardless of the urgency of the task to be merged acquired in step S201. Whether the level is "non-emergency", "normal” or “emergency”, it is determined that the combination of the tasks to be merged can be performed; if the load level of the current system acquired in step S201 is "medium load", it matches the example of Table 1.
  • the merging task execution policy when the urgency level of the task to be merged obtained in step S201 is "normal” or “emergency”, it is also determined that the merging of the tasks to be merged can be performed; if the load level of the current system acquired in step S201 is "High load", then matching to the merge task execution policy of the example in Table 1, only when the urgency level of the task to be merged acquired in step S201 is "emergency", it is determined that the merge of the task to be merged can be performed; When the load level of the current system acquired by S201 is "saturated", it matches the merge task execution of the example in Table 1.
  • step S201 regardless of whether the urgency level of the task to be merged obtained in step S201 is "non-emergency", "normal” or “emergency", it is determined that the merging of the tasks to be merged cannot be performed; if the load level of the current system acquired in step S201 The "high load” is matched to the merge task execution policy of the example in Table 1. When the urgency level of the task to be merged acquired in step S201 is "non-emergency" or "normal”, it is also determined that the to-be-committed may not be executed.
  • step S201 If the load level of the current system acquired in step S201 is "medium load”, the matching task execution policy of the example in Table 1 is matched, and when the urgency level of the task to be merged acquired in step S201 is "non-emergency" It is also determined that the merging of the tasks to be merged cannot be performed.
  • the merge task execution module is a functional unit that performs task merge, and one or more such functional units may exist in the child table server.
  • the merged task execution module is idle, the task can be performed, and the task is actively requested from the scheduler, or the active report is idle, and the scheduler is assigned to the task. Therefore, according to step S201 and step S202, if it is determined that the task to be merged can be executed, that is, it is determined that the merge can be performed on the task to be merged, The merged task is sent to the merge task execution module, and the merge task execution module performs task merge.
  • the to-be-combined task is returned to the waiting queue, and waits for an appropriate timing to be taken out from the waiting queue to perform task combining.
  • the sub-table server is not suitable for performing task consolidation for a relatively long period of time, for example, if the load of the current system is in a long period of time.
  • the merging of tasks If, during the period of the above example, the tasks in the waiting queue cannot be performed according to the merge task execution policy before the queue is fetched frequently from the waiting queue, it is obviously a waste of resources.
  • step S201 and step S202 After the merge task execution policy according to the example in Table 1 cannot perform the task merge and returns to the waiting queue in the waiting queue, the task to be merged may be frozen, so that the task to be merged is within the set freeze time. It is in a state in which the waiting queue cannot be taken out.
  • the size of the freeze time setting may be set by the user according to the load level of the current system and/or the expected duration of the urgency level of the task to be merged, which is not particularly limited in the present invention.
  • a timer may be started to time the set freeze time so that after the set freeze time is over,
  • the merge task execution policy according to the table 1 cannot perform the task merge and the task to be merged in the wait queue is changed to the state in which the wait queue can be taken out, thereby reevaluating whether the merge task can perform the merge.
  • the freeze time is set based on the length of the expected duration of the current system's load level, it is also impossible to exclude that the freeze time setting exceeds the expected duration of the current system load level. For example, if the load level of the current system or/and the urgency level of the task to be merged changes, thereby performing the merge task execution strategy as exemplified in Table 1, the previous example according to Table 1 And the task execution strategy can not perform the merged to-be-merge task, and the merge can be re-executed. At this time, if the mechanical merge is waited for the set freeze time to end, the task merge is performed, which is also a waste of resources for the system.
  • a change occurs in the load level of the current system, that is, if the load level of the current system is decreased from “saturated” to "high load”, “medium load” or “low load”, Then, in step S201 and step S202, the merge task execution policy according to the table 1 may not perform task merge, and the task to be merged in the wait queue is changed to a state in which the wait queue can be taken out; or, if the load of the current system is If the level is lowered from "high load” to "medium load” or "low load”, the merge task execution strategy according to Table 1 in step S201 and step S202 can not perform task merge and return to the task to be merged in the wait queue.
  • the change of the urgency level of the task to be merged that is, if the merged task execution policy according to the table 1 in step S201 and step S202 cannot perform the task merging and returns to the waiting queue If the urgency level of the merging task is raised from “non-emergency" to "normal” or “emergency", the task to be merged is changed to a state in which the waiting queue can be taken out; or, if in step S201 and step S202, according to Table 1
  • the example merge task execution policy cannot perform task merge and the urgency level of the task to be merged in the wait queue is raised from "normal” to "emergency", and the task to be merged is changed to a state in which the wait queue can be taken out.
  • the merge task execution strategy according to Table 1 is performed in steps S201 and S202 If the urgency level of the task to be merged in the waiting queue cannot be performed and the urgency level of the task to be merged is higher than that of the other tasks to be merged, the merged task execution policy according to the example in Table 1 in step S201 and step S202 cannot be executed.
  • the tasks to be merged and returned to the waiting queue are changed to a state in which the waiting queue can be taken out.
  • the method provided by the embodiment of the present invention can be based on the prior art without considering the load or the state of the system (for example, the system is relatively busy and/or the system load is relatively high) and the merge of the tasks to be merged is still performed without limitation.
  • the system resource occupation situation is used to schedule the data merge task, so that the data merge task is automatically delayed when the system load is high, and is automatically executed when the system is under low load, thereby reducing the impact of the execution task merge on the access service, and allowing the task to be merged.
  • the merger is performed automatically and in a timely manner at the right time.
  • FIG. 3 it is a structural diagram of a data merging device of a key value database according to an embodiment of the present invention. For the convenience of description, only parts related to the embodiment of the present invention are shown.
  • the data combining apparatus of the key value database illustrated in Fig. 3 includes an obtaining module 301, a determining module 302, and a processing module 303, wherein:
  • the obtaining module 301 is configured to acquire a load level of the current system and an urgency level of the task to be merged.
  • the merge task of the key value database is related to the load of the current system, for example, when the system is busy and/or the system load is relatively high, the unrestricted execution of the key database merge task will increase the system burden and affect the data access service. Performance.
  • whether or not the merge task of the key-value database is executed is also related to the urgency of the task to be merged. Therefore, in this embodiment, the level of the current system and the level of urgency of the task to be merged may be first acquired by the level obtaining module 301, where the load level of the current system includes but is not limited to low load, medium load, high load, and There are several types of saturation, such as non-emergency, normal and emergency.
  • the determining module 302 is configured to use, according to the load level of the current system and the emergency process of the task to be merged, in the embodiment, the combined task execution policy is pre-defined according to the load level of the system and the urgency level of the task to be merged.
  • the principle of determining whether to perform task consolidation includes the following examples: the higher the load level of the current system, the higher the urgency task to be merged, that is, the higher the urgency level is to be performed. Merging, when the current system's load level is saturated, it is not allowed to perform any tasks to be merged, that is, it is not allowed to perform the merge of any tasks to be merged.
  • the execution policy of the merged task may also be interpreted as: when the urgency level of the task to be merged is determined, the higher the load level of the current system, the task to be merged is allowed to execute.
  • the lower the probability of the sum the lower the load level of the current system, the higher the probability that the task to be merged is allowed to perform the merge; or the urgency level of the task to be merged when the load level of the current system is determined
  • the higher the degree ie, the more urgent the task to be merged needs to perform the merge
  • the lower the level of urgency of the task to be merged ie, the task to be merged needs to perform the merge
  • the more relaxed the degree the higher the probability that the task to be merged is not allowed to perform the merge; when the load level of the current system is the lowest, for example, when the load level of the current system is low load, the urgency level can be merged.
  • Task execution merge When the current system's load level is the highest, for example, when the current system's load level is saturated, it is not allowed to perform merging for any urgency level to be merged task; when the urgency level of the task to be merged is the highest, for example , the urgency level of the task to be merged is urgent, unless the current system
  • the load level is the highest, for example, the load level of the current system is saturated, otherwise, the merge is allowed to be performed on the task to be merged whose emergency level is urgent.
  • the current system's load levels include, but are not limited to, low load, medium load, high load, and saturation
  • the urgency levels of the tasks to be combined include, but are not limited to, non-emergency, normal, and emergency.
  • the merge task execution policy provided in this embodiment may be as shown in Table 1 above.
  • the processing module 303 is configured to: if the determining module 302 determines to perform the to-be-committed task, send the to-be-combined task to the merged task execution module to perform task combining.
  • the merge task execution module is a functional unit that performs task merge, and one or more such functional units may exist in the child table server.
  • the merged task execution module is idle, the task can be performed, and the task is actively requested from the scheduler, or the active report is idle, and the scheduler is assigned to the task. Therefore, once the determining module 302 determines that the to-be-combined task can be performed, that is, it is determined that the merge can be performed on the to-be-combined task, the processing module 303 sends the to-be-combined task to the merged task execution module, which is executed by the merged task execution module. Task consolidation.
  • the processing module 303 may further return the to-be-combined task to the waiting queue, wait for a suitable timing, and wait for another time. Take out the queue to perform task consolidation.
  • each functional module is only an example, and the actual application may be configured according to requirements, for example, corresponding hardware requirements.
  • the convenience of the implementation of the software, and the above-mentioned function allocation is completed by different functional modules, that is, the internal structure of the data combining device of the key value database is divided into different functional modules to complete all or part of the functions described above.
  • the corresponding functional modules in this embodiment may be implemented by corresponding hardware, or may be executed by corresponding hardware.
  • the foregoing level obtaining module may have the foregoing acquisition current
  • the hardware of the load level of the system and the urgency level of the task to be merged such as a level acquirer, may also be a general processor or other hardware device capable of executing a corresponding computer program to perform the aforementioned functions;
  • An emergency program, such as a determiner, that performs the foregoing load level according to the current system and a task to be merged, such as a determiner, may also be a general processor or other hardware device capable of executing a corresponding computer program to perform the aforementioned functions (each provided in this specification)
  • the above described principles can be applied to the embodiments).
  • the acquisition module 301 of the example of FIG. 3 may include a first acquisition unit 401, such as the data integration device of the key value database provided by another embodiment of the present invention as shown in FIG.
  • the first obtaining unit 401 is configured to perform a weighted summation of the usage of the resources included in the current system according to the weight, and assign a corresponding level to the load of the current system according to the value obtained by the weighted summation, where the system is Resource usage includes any combination of CPU usage, memory usage, network usage, and disk usage, including current load, medium load, high load, and saturation.
  • a central processing unit (CPU), a memory, a network, a disk, and the like all belong to resources included in the current system.
  • the first obtaining unit 401 may perform weighted summation according to the weight of each resource after acquiring the usage of the resources, and the weighted summation value is equivalent to scoring the current system load, for example, if the current system load score When L is used, the expression of L is as shown in the above formula (1). If the load of the system can be divided into low load, medium load, high load and saturation according to the level from low to high, the first obtaining unit 401 can load the current system according to the load score L of the system from low to high. The corresponding level is assigned in turn.
  • the first acquisition unit is when the load score of the system is in the range of 0 to 25, 25 to 50, 50 to 75, and 75 to 100.
  • 401 correspondingly sequentially assigns the load level of the current system to low load, medium load, high load, and saturation.
  • the acquiring module 301 may further include other functional modules or units for acquiring the urgency level of the task to be merged, and the obtaining manner thereof is in the drawing. 4
  • the device of the example is not limited.
  • the level obtaining module 301 of the example of FIG. 3 may also include a second obtaining unit 501, such as the data combining device of the key value database provided by another embodiment of the present invention as shown in FIG.
  • the second obtaining unit 501 is configured to linearly weight the factors affecting the urgency of the task to be merged according to the weight, and assign a corresponding level to the urgency of the task to be merged according to the value obtained by the linear weighted summation.
  • the factors affecting the urgency of the task to be merged include any combination of the file size in the sub-table, the number of files in the sub-table, and the expiration time of the oldest data in the sub-table, where the task to be merged Urgency levels include general, urgent, and mandatory.
  • factors affecting the urgency of the task to be merged include, but are not limited to, the file size in the sub-table, the number of files in the sub-table, and the overdue time of the oldest data in the sub-table. and many more. Since there are quantitative differences in different influencing factors, the second obtaining unit 501 can calculate the urgency score of the task to be merged by means of linear weighted summation. In this way, for each factor affecting the urgency of the task to be combined, the percentage is calculated according to the maximum value of each factor and weighted and summed. If the urgency score Srar of the task to be combined is expressed by the above formula (2) The urgency level of the task to be merged may be divided into normal, urgent, and mandatory, and the second obtaining unit 501 may score according to the urgency of the task to be merged.
  • the second obtaining unit 501 sequentially assigns the urgency level of the task to be combined to ordinary, urgent, and necessary .
  • the obtaining module 301 may further include other functional modules or units for acquiring the load level of the current system, and the obtaining manner thereof is illustrated in FIG. 5.
  • the device is not limited.
  • the urgency of the combined task is scored, considering the general state of each factor in the common scenario, however, in some scenarios, despite other emergency affecting the task to be merged
  • the score of the degree factor may still be relatively low, but when the factor affecting the urgency of the task to be merged is close to or reaches the maximum value, the overall impact of the urgency of the execution combination of the task to be merged begins to be nonlinear.
  • the ground is enlarged. Since the urgency of the merging task is evaluated by the formula (2), the linear weighted summation is used.
  • the second obtaining unit 501 still follows the formula (2).
  • the acquisition module 301 exemplified in FIG. 3 may also include a third acquisition unit 601, as shown in FIG. 6a, which is a key value database provided by another embodiment of the present invention. Data merge device.
  • a third obtaining unit 601 configured to perform non-linear weighted summation of each factor affecting the urgency of the task to be merged according to a weight, and assign a urgency to the task to be merged according to the value obtained by the weighted summation
  • the level of the urgency affecting the task to be merged includes any combination of the file size in the child table, the number of files in the child table, and the time of the oldest data in the child table, the task to be merged Urgency levels include non-emergency, general and emergency.
  • the obtaining module 301 may further include other functional modules or units for acquiring the load level of the current system, and the obtaining manner thereof is illustrated in FIG. 6a.
  • the device is not limited.
  • the third obtaining unit 601 illustrated in Fig. 6a may further include an amplifying unit 6011 and a summing unit
  • a data merging device for a key value database according to another embodiment of the present invention, wherein:
  • the amplifying unit 6011 is configured to expand the weight of the factor affecting the urgency of the task to be merged to the weight amplifying function.
  • a summation unit 6012 configured to weight the factors affecting the urgency of the task to be merged according to the new weight according to the new weight as a factor affecting the urgency of the task to be merged, according to the new weight a value corresponding to the urgency of the task to be merged, that is, a summation unit 6012 assigns a urgency to the task to be merged according to formula (3)
  • the level of responsibility including general, urgent, and must, and so on.
  • the determining module 302 illustrated in FIG. 4 to FIG. 6b may include a first determining unit 701, a second determining unit 702, and a third determining unit 703, as shown in FIG. 7a to FIG. 7d, which are provided by another embodiment of the present invention.
  • a data merging device for a key value database where:
  • the first determining unit 701 is configured to: if the urgency level of the task to be merged is urgent, when the load level of the current system is low load, medium load, or high load, determine to allow execution of the to-be-committed The merging of tasks.
  • the acquisition module 301 matches the merge task execution policy of the example in Table 1.
  • the load level of the current system acquired by the acquisition module 301 is "low load”.
  • the first determining unit 701 determines that the task to be merged is allowed to be executed, that is, determines that the merging is allowed to be performed on the task to be merged, or the load level of the current system acquired by the obtaining module 301 is “medium load”, and the first determining unit 701 determines
  • the task to be merged is allowed to be executed, that is, it is determined that the merging is allowed to be performed on the to-be-combined task, or the load level of the current system acquired by the obtaining module 301 is “high load”, and the first determining unit 701 determines that the to-be-committed is allowed to be executed.
  • the task ie determining to allow the merging of the tasks to be merged.
  • the second determining unit 702 is configured to: if the urgency level of the task to be merged is non-emergency, when the load level of the current system is greater than the medium load or the medium load, determine that the to-be-committed is not allowed to be performed. The task, that is, determines that the merging of the tasks to be merged is not allowed.
  • the acquisition module 301 matches the merge task execution policy of the example in Table 1.
  • the load level of the current system acquired by the acquisition module 301 is "medium load”.
  • the second determining unit 702 determines that the to-be-combined task is not allowed to be executed, that is, it is determined that the merging is not allowed to be performed on the to-be-joined task, or the load level of the current system acquired by the obtaining module 301 is “high load”, and the second determination is performed.
  • the unit 702 determines that the task to be merged is not allowed to be executed, that is, it is determined that the merge is not allowed to be performed on the task to be merged, or the load level of the current system acquired by the obtaining module 301 is "saturated", and the second determining unit 702 determines that the task is not allowed.
  • the tasks to be merged are executed, that is, it is determined that the merge is not allowed to be performed on the tasks to be merged.
  • the third determining unit 703 is configured to: if the urgency level of the task to be merged is normal, when the load level of the current system is higher than a high load or a high load, determine that the performing is not allowed To be merged.
  • the merge task execution policy of the example of Table 1 is matched, and the load level of the current system acquired by the obtaining module 301 is "
  • the third determining unit 703 determines that the to-be-combined task is not allowed to be executed, that is, it is determined that the merging of the to-be-combined task is not allowed.
  • the data merging apparatus of the key value database illustrated in Figures 4 through 6b may include a task freeze module 801, such as the data merging apparatus of the key value database provided by another embodiment of the present invention, as shown in Figures 8a to 8d.
  • the task freeze module 801 is configured to freeze the to-be-consolidated task, so that the to-be-combined task is in a state in which the waiting queue cannot be taken out within a set freeze time.
  • the sub-table server should not perform task consolidation for a relatively long period of time, for example, if the current system load is in a long period of time
  • the task freezing module 801 may freeze the to-be-consolidated task to make the waiting The merge task is in a state in which the wait queue cannot be fetched within the set freeze time.
  • the size of the freeze time setting may be set by the user according to the load level of the current system and/or the expected duration of the urgency level of the task to be merged, which is not specifically limited by the present invention.
  • the data merging device of the key value database illustrated in Figures 8a to 8d may include a first defrosting module 901, such as the data merging device of the key value database provided by another embodiment of the present invention as shown in Figures 9a to 9d.
  • the first defrosting module 901 is configured to start a timer to time the set freezing time, so that after the set freezing time ends, the to-be-combined task is changed to a state in which the waiting queue can be taken out. . Since the freeze time is set based on the length of the expected duration of the current system's load level, it is also impossible to exclude that the freeze time setting exceeds the expected duration of the current system load level.
  • the data merging device of the key value database illustrated in FIGS. 8a to 8d may also include a second defrosting module 1001, a third defrosting module 1002, or a fourth defrosting module 1003, as shown in FIG. 10a to FIG. 10d.
  • a data merging device for a key value database provided by another embodiment, wherein:
  • the second defrosting module 1001 is configured to: if the load level of the current system decreases from saturation to high load, medium load or low load, from high load to medium load or low load, or from medium load to low load, The task to be merged is changed to a state in which the waiting queue can be taken out.
  • the third defrosting module 1002 is configured to change the to-be-combined task to be able to take out the waiting queue if the urgency level of the task to be merged rises from non-emergency to normal or urgent, or from ordinary to urgent. status.
  • the fourth defrosting module 1003 is configured to: when all the tasks to be merged are frozen in the waiting queue and the merging task execution module is in an idle state, if the urgency level of the task to be merged is higher than the urgency level of other tasks to be merged And changing the to-be-combined task to a state in which the waiting queue can be taken out.
  • the to-be-combined task is sent to the merged task execution module to perform a merge task.
  • the program may be stored in a computer readable storage medium, and the storage medium may include: Read Only Memory (ROM), Random Access Memory (RAM), disk or optical disk.
  • ROM Read Only Memory
  • RAM Random Access Memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

La présente invention concerne un procédé et un dispositif de fusionnement de données d'une base de données de valeurs de clés, afin de commander une tâche de fusionnement de données et de manière à empêcher une augmentation de la charge imposée au système du fait de la simultanéité d'un grand nombre de tâches de fusionnement de données. Le procédé consiste à obtenir un niveau de charge d'un système courant et un niveau d'urgence d'une tâche à fusionner; conformément au niveau de charge du système courant et du niveau d'urgence de la tâche à fusionner, à apparier une règle d'exécution de fusionnement de tâches afin de déterminer s'il est permis d'exécuter la tâche à fusionner; s'il est déterminé qu'il est permis d'exécuter la tâche à fusionner, à envoyer la tâche à fusionner à un module d'exécution de fusionnement de tâches afin d'exécuter le fusionnement de tâches. Le procédé proposé par la présente invention permet de commander la tâche de fusionnement de données en fonction du taux d'occupation des ressources du système afin de retarder automatiquement la tâche de fusionnement des données lorsque la charge du système est élevée, cela permettant de réduire l'effet de l'exécution de fusionnement de tâches sur l'accès au service, et de réduire la charge imposée au système.
PCT/CN2012/077314 2012-06-21 2012-06-21 Procédé et dispositif de fusionnement de données d'une base de données de valeurs de clés WO2013189063A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201280000718.XA CN103649948A (zh) 2012-06-21 2012-06-21 键值数据库的数据合并方法和装置
PCT/CN2012/077314 WO2013189063A1 (fr) 2012-06-21 2012-06-21 Procédé et dispositif de fusionnement de données d'une base de données de valeurs de clés

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2012/077314 WO2013189063A1 (fr) 2012-06-21 2012-06-21 Procédé et dispositif de fusionnement de données d'une base de données de valeurs de clés

Publications (1)

Publication Number Publication Date
WO2013189063A1 true WO2013189063A1 (fr) 2013-12-27

Family

ID=49768044

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2012/077314 WO2013189063A1 (fr) 2012-06-21 2012-06-21 Procédé et dispositif de fusionnement de données d'une base de données de valeurs de clés

Country Status (2)

Country Link
CN (1) CN103649948A (fr)
WO (1) WO2013189063A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105489216A (zh) * 2016-01-19 2016-04-13 百度在线网络技术（北京）有限公司 语音合成系统的优化方法和装置

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101894163A (zh) * 2010-07-23 2010-11-24 中兴通讯股份有限公司 一种针对性能数据采集系统的数据库操作调度方法及装置

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101894163A (zh) * 2010-07-23 2010-11-24 中兴通讯股份有限公司 一种针对性能数据采集系统的数据库操作调度方法及装置

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
FEI, JIANGTAO ET AL.: "Technology Summarization of Mass-data Management System Based on Table Structure", COMPUTER AND MODERNIZATION, no. 1, January 2010 (2010-01-01), pages 166 - 169 *
WANG, YINGDONG ET AL.: "The Research for Load Balancing Technology of Bigtable System", NETWORK AND COMPUTER SECURITY, no. 2, February 2009 (2009-02-01), pages 41 - 43 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105489216A (zh) * 2016-01-19 2016-04-13 百度在线网络技术（北京）有限公司 语音合成系统的优化方法和装置
US10242660B2 (en) 2016-01-19 2019-03-26 Baidu Online Network Technology (Beijing) Co., Ltd. Method and device for optimizing speech synthesis system

Also Published As

Publication number Publication date
CN103649948A (zh) 2014-03-19

Similar Documents

Publication Publication Date Title
US11593404B2 (en) Multi-cluster warehouse
JP5744707B2 (ja) メモリ使用量照会ガバナのためのコンピュータ実装方法、コンピュータ・プログラム、およびシステム（メモリ使用量照会ガバナ）
KR101721892B1 (ko) 쿼리 관리
US9286116B2 (en) Multiple core real-time task execution
Luo et al. On performance stability in LSM-based storage systems (extended version)
JP5733680B2 (ja) コマンド実行装置、コマンド実行システム、コマンド実行方法およびコマンド実行プログラム
JP5949937B2 (ja) トランザクションシステム
GB2533116A (en) Query dispatching system and method
CN110781244B (zh) 用于对数据库的并发操作进行控制的方法及装置
JP5686034B2 (ja) クラスタシステム、同期制御方法、サーバ装置および同期制御プログラム
US10148531B1 (en) Partitioned performance: adaptive predicted impact
WO2022062184A1 (fr) Procédé hautement concurrentiel d'interrogation, terminal intelligent et support de stockage
US20170357681A1 (en) Dynamic in-flight database request throttling
JP5331050B2 (ja) データ同期システム、データ同期方法、情報処理装置、情報処理方法、およびプログラム
US10033620B1 (en) Partitioned performance adaptive policies and leases
WO2013189063A1 (fr) Procédé et dispositif de fusionnement de données d'une base de données de valeurs de clés
WO2023066070A1 (fr) Procédé et appareil de commande d'instruction sql
JP6046523B2 (ja) インメモリ型分散データベース、データ分散方法及びプログラム
US20080282247A1 (en) Method and Server for Synchronizing a Plurality of Clients Accessing a Database
JP6459654B2 (ja) イベントドリブンシステム、情報処理装置、イベントドリブンプログラムおよびイベントドリブン方法
US20210227018A1 (en) Lazy lock queue reduction for cluster group changes
CN113986962A (zh) 排行榜生成方法、装置、设备及存储介质
WO2013082743A1 (fr) Procédé et dispositif permettant d'améliorer la performance de concurrence d'un système de stockage d'objets distribué
Xia et al. An optimized load balance based on data popularity on HBASE
JP2013077046A (ja) データ格納制御装置、データ格納制御プログラムおよびデータ格納制御方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12879170

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12879170

Country of ref document: EP

Kind code of ref document: A1