CN115033585B - Data merging processing method and device for target database - Google Patents
Data merging processing method and device for target database Download PDFInfo
- Publication number
- CN115033585B CN115033585B CN202210947846.0A CN202210947846A CN115033585B CN 115033585 B CN115033585 B CN 115033585B CN 202210947846 A CN202210947846 A CN 202210947846A CN 115033585 B CN115033585 B CN 115033585B
- Authority
- CN
- China
- Prior art keywords
- partition
- version
- data
- parameter
- version number
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2329—Optimistic concurrency control using versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
Abstract
The embodiment of the specification provides a data merging processing method and device for a target database, wherein the target database comprises a plurality of partition groups, any partition group comprises a plurality of data partitions, and is provided with a first version parameter and a second version parameter, and the method comprises the following steps: in response to detecting that each data partition of any target partition group in the plurality of partition groups completes data merging, updating a parameter value of a first version parameter of the target partition group from a current first version number to a second version number; wherein the second version number is higher than the first version number; determining whether the parameter values of the first version parameters of the plurality of partition groups are all second version numbers; if the determination result is yes, updating the parameter value of the second version parameter of each of the plurality of partition groups from the current first version number to the second version number.
Description
Technical Field
The embodiment of the specification relates to the technical field of databases, in particular to a data merging processing method and device for a target database.
Background
Some existing databases support data partitioning (partition) and partition groups (partiongroup), and a single partition group may include several data partitions. The database generally merges the data in the respective data partitions of all partition groups at regular intervals to form a globally incremented merged version number for data verification. During the merging process, the merging progress of the multiple data partitions is generally different, and the data partitions may be transferred between different partition groups.
Therefore, a reasonable and reliable scheme for ensuring that the incremental merging version number is not rolled back is urgently needed.
Disclosure of Invention
The embodiment of the specification provides a data merging processing method and device for a target database, which can ensure that the incremental merging version number is not returned.
In a first aspect, an embodiment of the present specification provides a data merging processing method for a target database, where the target database includes multiple partition groups, where any partition group includes multiple data partitions, and is provided with a first version parameter and a second version parameter, the method includes: in response to detecting that each data partition of any target partition group in the plurality of partition groups completes data merging, updating a parameter value of a first version parameter of the target partition group from a current first version number to a second version number; wherein the second version number is higher than the first version number; determining whether the parameter values of the first version parameters of the plurality of partition groups are the second version numbers; if the determination result is yes, updating the parameter value of the second version parameter of each of the plurality of partition groups from the first version number to the second version number.
In some embodiments, the plurality of partition groups comprises a first partition group and a second partition group, the first partition group comprising a first data partition that has not been data merged and having associated therewith a transfer task to transfer the first data partition to the second partition group; and the method further comprises: if the version number of the first data partition is not lower than the parameter value of the second version parameter of the second partition group, performing the transfer task to transfer the first data partition from the first partition group to the second partition group.
In some embodiments, after transferring the first data partition from the first partition group to the second partition group, further comprising: and if the parameter value of the first version parameter of the second partition group is the second version number, returning the parameter value of the first version parameter to the first version number.
In some embodiments, the method further comprises: and if the version number of the first data partition is lower than the parameter value of the second version parameter of the second partition group, suspending the execution of the transfer task until the first data partition finishes data merging.
In some embodiments, each data partition of any partition group is provided with a third version parameter; and detecting that each data partition of any target partition group in the plurality of partition groups completes data merging, specifically comprising: if it is detected that any data partition in the target partition group completes data merging, updating a parameter value of a third version parameter of the any data partition from the first version number to the second version number; and when the parameter values of the third version parameters of the data partitions of the target partition group are the second version numbers, determining that the data partitions of the target partition group complete data merging.
In some embodiments, the target database includes a merge controller.
In some embodiments, the method further comprises: sending, via the merge controller, a data merge request to the plurality of partition groups, respectively, the data merge request including the second version number; such that the plurality of partition groups merge data within respective data partitions.
In some embodiments, the determining whether the parameter values of the first version parameter of each of the plurality of partition groups are all the second version number comprises: and respectively sending parameter value acquisition requests aiming at the first version parameters to the plurality of partition groups through the merging controller, receiving acquisition results respectively returned by the plurality of partition groups, and determining whether the parameter values of the first version parameters of the plurality of partition groups are the second version numbers according to the received acquisition results.
In some embodiments, said updating the parameter value of the second version parameter of each of the plurality of partition groups from the first version number to the second version number comprises: sending, via the merge controller, a parameter value update request for a second version parameter to the plurality of partition groups, respectively, so that the plurality of partition groups each update a parameter value of their second version parameter from the first version number to the second version number.
In some embodiments, after sending the parameter value update request for the second version parameter to the plurality of partition groups, respectively, the method further comprises: receiving the updating results respectively returned by the plurality of subarea groups; when the received updating results are all updating success, determining that the combination is completed; when at least one of the updating results is updating failure, after the interval is set for a long time, continuing to execute the step of determining whether the parameter values of the first version parameters of the plurality of partition groups are the second version numbers; and the updating result belonging to the updating failure is generated by the corresponding partition group in response to the parameter value of the first version parameter returning to the first version number.
In a second aspect, an embodiment of the present specification provides a data merging processing apparatus for a target database, where the target database includes a plurality of partition groups, where any partition group includes a plurality of data partitions, and is provided with a first version parameter and a second version parameter, the apparatus includes: a first updating unit configured to update a parameter value of a first version parameter of any target partition group of the plurality of partition groups from a current first version number to a second version number in response to detecting that each data partition of the target partition group completes data merging; wherein the second version number is higher than the first version number; a determining unit configured to determine whether parameter values of the first version parameters of each of the plurality of partition groups are all the second version numbers; a second updating unit configured to update a parameter value of a second version parameter of each of the plurality of partition groups from the first version number to the second version number if a result of the determination by the determining unit is yes.
In a third aspect, the present specification provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed in a computer, the computer is caused to execute the method described in any implementation manner in the first aspect.
In a fourth aspect, the present specification provides a computing device, including a memory and a processor, where the memory stores executable code, and the processor executes the executable code to implement the method described in any implementation manner of the first aspect.
In a fifth aspect, the present specification provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the method described in any implementation manner of the first aspect.
In the solution provided by the above embodiment of the present specification, the target database includes a plurality of partition groups, where any partition group includes a plurality of data partitions, and is provided with a first version parameter and a second version parameter. In this scheme, in response to detecting that each data partition of any target partition group in the plurality of partition groups completes data merging, the parameter value of the first version parameter of the target partition group may be updated from the current first version number to the second version number. Wherein the second version number is higher than the first version number. It should be noted that the parameter value of the first version parameter may indicate whether data merging is completed for each data partition of the corresponding partition group. For example, when the parameter value is still the first version number, it may indicate that the data partitions do not complete data merging; when the parameter value is updated to a second version number, it may indicate that each data partition completes data merging. In addition, whether the parameter values of the first version parameters of the plurality of partition groups are all the second version numbers can be determined. If the determination result is yes, it may be considered that the data partitions of the multiple partition groups all complete data merging, so that the parameter value of the second version parameter of each of the multiple partition groups may be updated from the current first version number to the second version number. It should be understood that the updated parameter value of the second version parameter, i.e. the second version number, is the merged version number after the current round of merging is finished.
By adopting the mode of maintaining the double-version parameters in the partition groups, whether the merging progress of each data partition is the same in the merging process and whether the data partitions are transferred among different partition groups or not can be ensured as much as possible, the parameter value of the second version parameter can be increased from the current first version number to the second version number when all the data partitions are merged, and therefore, the increment of the merged version number can be ensured without returning.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments disclosed in the present specification, the drawings required to be used in the description of the embodiments will be briefly described below, it is obvious that the drawings in the following description are only the embodiments disclosed in the present specification, and it is also possible for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
FIG. 1 is a schematic diagram of a data merging process for data partitions;
FIG. 2 is a schematic diagram of the constituent structure of the target database;
FIG. 3 is a schematic diagram of a data consolidation process;
FIG. 4 is a flow diagram of one embodiment of a data merge processing method for a target database;
FIG. 5 is a schematic diagram of a data consolidation process;
FIG. 6 is a schematic diagram of a data partition migration process;
FIG. 7 is a schematic illustration of the effect of the display of partition group 1 and partition group 2;
FIG. 8 is a schematic diagram of a data partition migration process;
fig. 9 is a schematic structural diagram of a data merge processing apparatus for a target database.
Detailed Description
The present specification will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. The described embodiments are only some of the embodiments of the present specification, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step are within the scope of the present application.
It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings. The embodiments and features of the embodiments in the present description may be combined with each other without conflict.
In practice, a database supporting data partitions and partition groups typically includes a plurality of partition groups, any of which includes a number of data partitions. Any data partition may be as shown in FIG. 1, including delta data and baseline data. Fig. 1 is a schematic diagram of a data merging process of data partitions.
In one example, the storage architecture employed by the database may be an LSM-Tree (Log Structured Merge Tree), and the data in the data partition may be divided into incremental data and baseline data based on the LSM-Tree.
The baseline data is generally data that is not changed any more, and the incremental data is generally updated in real time and is continuously expanded. When merging is required, the delta data and the baseline data may be merged to form new baseline data, as shown in fig. 1. The new baseline data may be written into the corresponding data partition, and the original incremental data and baseline data in the data partition, that is, the source of the new baseline data, may be deleted after the new baseline data is formed, and specifically, may be deleted after the new baseline data is written into the data partition. Therefore, the purposes of reducing storage space, accelerating reading and verifying data can be achieved.
Additionally, the database may be a distributed database. The plurality of partition groups may be distributed across a plurality of servers of the database. To achieve load balancing, the database may have the functionality to migrate data partitions between different partition groups.
As mentioned above, the database typically merges the data in the respective data partitions of its partition groups at intervals to form a globally incremented merged version number for data verification. During the merging process, the merging progress of the multiple data partitions is generally different, and the data partitions may be transferred between different partition groups.
Based on this, the embodiments of the present specification provide a data merging processing method for a target database, which can ensure that incremental merging version numbers do not fall back.
Next, the composition structure and the data merging process of the target database will be described with reference to fig. 2 and 3.
Reference is first made to fig. 2, which is a schematic illustration of the structure of the target database. The target database includes a plurality of partition groups, such as partition group 1 and partition group 2 shown in FIG. 2. Any of the plurality of partition groups may include a number of data partitions. As shown in FIG. 2, partition group 1 may include a data partition 1, a data partition 2, and a data partition 3, and partition group 2 may include a data partition 4. Any of the several data partitions may be as described above, including delta data and baseline data.
It should be noted that only two partition groups are shown in fig. 2 for convenience of description. It should be understood that the target database may include more partition groups.
Any one of the plurality of partition groups may be provided with a dual version parameter. One of the two version parameters may be used as an auxiliary parameter in the merging process, and a parameter value thereof may indicate whether each data partition of the corresponding partition group completes data merging. For convenience of description, the version parameter may be referred to as a first version parameter. In practice, the parameter value of the first version parameter may allow for rollback. For example, after the parameter value of the first version parameter is updated from the original version number to the new version number, the parameter value of the first version parameter may be rolled back from the new version number to the original version number for some reason (e.g., migrating a data partition having the version number of the original version number into a corresponding partition group, etc.).
The other version parameter of the dual version parameter may function as a parameter for recording the merged version number after the merging ends. For convenience of description, the other version parameter may be referred to as a second version parameter. In practice, the parameter value of the second version parameter cannot be backed off, and the parameter value of the second version parameter is less than or equal to the parameter value of the first version parameter.
The parameter name of the first version parameter may be current _ merge _ version as shown in fig. 2. The parameter name of the second version parameter may be determined _ merge _ version as shown in fig. 2. The current _ merge _ version and the determined _ merge _ version current parameter value 4 may be a merge version number after the previous round of data merging is finished. It should be noted that current _ merge _ version and determined _ merge _ version are only exemplary parameter names, the parameter value 4 is only an exemplary parameter value, and the first version parameter and the second version parameter may have other parameter names and parameter values, which is not limited herein.
The target database may merge data within the data partitions of each of partition group 1 and partition group 2 at intervals. For any partition group of the partition group 1 and the partition group 2, when each data partition of the any partition group completes data merging, the parameter value of current _ merge _ version of the any partition group may be updated from the current merging version number 4 to a new merging version number 5. Moreover, the target database may update the parameter value of the determined _ merge _ version of each of partition group 1 and partition group 2 from 4 to 5 when the parameter value of the current _ merge _ version of each of partition group 1 and partition group 2 is 5.
Optionally, the target database may further include a merge controller as shown in fig. 2, and the target database may perform merge control for partition group 1 and partition group 2 using the merge controller. The merge controller may be a unit or a device for initiating merge and determining end of merge, and may also be referred to as a merge control unit or a merge control device.
Specifically, when merging is needed, the merge controller may send a data merge request Q1 to partition group 1 and partition group 2, respectively, as shown in fig. 3. The data merge request Q1 may include a new merge version number 5. Fig. 3 is a schematic diagram of a data merging process.
After that, the partition group 1 may respond to the data merge request Q1, merge the data in the data partition 1, the data partition 2, and the data partition 3, and update the parameter value of current _ merge _ version of the group from 4 to 5 when all the three data partitions complete data merge. The partition group 2 may merge the data in the data partition 4 in response to the data merge request Q1, and update the parameter value of current _ merge _ version of the group from 4 to 5 when the data merge of the data partition 4 is completed.
The merge controller may periodically check whether the parameter value of current _ merge _ version for partition group 1 and partition group 2 is both 5. When detecting that the parameter values of current _ merge _ version of each of partition group 1 and partition group 2 are both 5, the merge controller may consider that the data partitions of each of partition group 1 and partition group 2 both complete data merging, and may send a parameter value update request Q2 for determined _ merge _ version to partition group 1 and partition group 2.
Partition group 1 and partition group 2 may then update the parameter value of the respective determined _ merge _ version from 4 to 5, respectively, in response to the parameter value update request Q2. After the parameter values of the determined _ merge _ version of the partition group 1 and the partition group 2 are both successfully updated to 5, the merge controller may determine that the merge is completed, that is, determine that the merge is completed.
By adopting the mode of maintaining double-version parameters in the partition groups, whether the merging progress of each data partition is the same in the merging process and whether the data partitions are transferred among different partition groups or not can be ensured as much as possible, and the parameter value of the determined _ merge _ version is increased from the current merging version number 4 to the new merging version number 5 when all the data partitions are merged, so that the merging version number can be ensured not to be increased back.
The specific steps of the above method are described below with reference to specific examples.
Referring to FIG. 4, a flow 400 of one embodiment of a data merge processing method for a target database is shown. The target database comprises a plurality of partition groups, wherein any partition group comprises a plurality of data partitions, and is provided with a first version parameter and a second version parameter. For the explanation of the first version parameter and the second version parameter, refer to the related description in the foregoing, and are not repeated here.
The execution subject of the method can be a target database, or a device in the target database responsible for data merging processing, and the like. The method comprises the following steps:
in step 406, if the determination result is yes, the parameter value of the second version parameter of each of the plurality of partition groups is updated from the first version number to the second version number.
In this embodiment, the data in the data partitions of the plurality of partition groups may be merged. Specifically, data merging may be performed for the plurality of partition groups at regular intervals.
In the process of merging the data in the data partitions of the multiple partition groups, it may be detected whether the data partitions of any target partition group in the multiple partition groups all complete data merging, and when it is detected that the data partitions of the target partition group all complete data merging, the parameter value of the first version parameter of the target partition group is updated from the current first version number to the second version number by performing step 402. Wherein the second version number is higher than the first version number. It should be understood that the second version number is a new merged version number. The first version number may be a merged version number after the merging of the previous round of data is finished.
In one example, each data partition of any partition group may be provided with a third version parameter. If it is detected that any data partition in the target partition group completes data merging, the parameter value of the third version parameter of the any data partition can be updated from the current first version number to the second version number. When the parameter values of the third version parameters of the data partitions of the target partition group are the second version numbers, it can be determined that the data merging is completed for each data partition of the target partition group.
Alternatively, step 404 may be performed periodically. In step 404, it may be determined whether the parameter values of the first version parameters of the plurality of partition groups are all the second version numbers. For example, whether the parameter values of the first version parameters of the plurality of partition groups are all the second version numbers may be detected by polling the plurality of partition groups.
If the result of the determination in step 404 is yes, it may be considered that the data partitions of the partition groups all complete data merging, so that the parameter values of the second version parameters of the partition groups may be updated from the first version number to the second version number by performing step 406. After the parameter values of the second version parameters of the plurality of partition groups are all successfully updated to the second version numbers, the completion of the merging in the current round can be judged.
In the scheme provided by the embodiment corresponding to fig. 4, a mode of maintaining dual version parameters in partition groups is adopted, and no matter whether the merging progress of each data partition is the same in the merging process and whether the data partition is transferred between different partition groups, it can be ensured as much as possible that the parameter value of the second version parameter is increased from the current first version number to the second version number when all the data partitions are merged, so that it can be ensured that the merging version number is increased without rollback.
In one embodiment, the target database may further include a merge controller as described above, and the merge controller may be used to perform merge control on the plurality of partition groups.
Next, a specific data merging process is described with reference to fig. 5. Fig. 5 is a schematic diagram of a data merging process. The data merging process comprises the following steps:
step 502, the merging controller sends data merging requests to the plurality of partition groups respectively, wherein the data merging requests comprise a second version number;
step 504, merging the data in the respective data partitions by the plurality of partition groups;
step 506, when the data merging of each data partition of the plurality of partition groups is completed, updating the parameter value of each first version parameter from the current first version number to a second version number;
step 508, the merging controller sends parameter value obtaining requests for the first version parameters to the plurality of partition groups respectively;
step 510, the plurality of partition groups respectively return the obtained results to the merge controller;
step 512, the merging controller determines whether the parameter values of the respective first version parameters of the plurality of partition groups are all second version numbers according to the received acquisition result;
step 514, in response to the determination result being yes, the merge controller respectively sends parameter value update requests for the second version parameters to the plurality of partition groups;
step 516, the aforesaid multiple partitions perform updating operation of updating the parameter value from the current first version number to the second version number for the respective second version parameter;
step 518, the plurality of partition groups respectively return update results to the merge controller;
step 520, the merging controller determines that merging is completed when all the received updating results are updating successfully;
in step 522, the merge controller continues to execute step 508 after setting a time interval when at least one of the update results is failed to update.
The data partitions of the plurality of partition groups may be all provided with the third version parameter. The plurality of partition groups may update the parameter value of the third version parameter of each arbitrary data partition from the current first version number to the second version number when detecting that the data merging of each arbitrary data partition is completed. Based on this, in step 506, the plurality of partition groups may consider that the respective data partitions complete data merging when the parameter values of the third version parameters of the respective data partitions are all the second version numbers, so as to update the parameter values of the respective first version parameters from the current first version numbers to the second version numbers.
It should be noted that after the step 512 is executed, if the determination result of the step 512 is yes, the step 514 may be executed next. If the determination result in step 512 is negative, after the time interval can be set, continue to execute step 508. The set time period may be set according to actual requirements, and is not limited herein.
In practice, after the parameter value of the first version parameter of the partition group is increased to the second version number, the parameter value may be moved back to the original first version number for some reason (for example, a data partition with the first version number is migrated into the partition group, etc.).
In order to cope with the situation that the parameter value of the first version parameter may fall back from the second version number to the first version number and ensure that the parameter value of the second version parameter of the partition group is increased from the first version number to the second version number only after all the data partitions of the partition group complete data merging, in step 516, when any partition group in the plurality of partition groups performs the updating operation, it may be determined whether the parameter value of the first version parameter of the partition group is the second version number. If the parameter value of the own first version parameter is confirmed to be the second version number, the parameter value of the own second version parameter can be updated from the current first version number to the second version number, and then an updating result used for representing the successful updating can be generated. If the parameter value of the own first version parameter is determined not to be the second version number, for example, the first version number is returned, the update failure can be determined, and an update result for representing the update failure is generated.
The data merging process is described above. Next, the migration processing of the data partitions in the data merging process will be described.
In practice, during data consolidation, certain data partitions may need to be transferred between different partition groups. If the data merging function and the data partition transferring function of the target database have a large mutual influence, the system operation may be unstable.
According to the scheme provided by the embodiment of the specification, the mutual influence between the data merging function and the data partition transferring function can be reduced as much as possible by adopting the double-version parameters, so that the system stability is obviously improved. For example, in the data merging process, when a data partition needs to be transferred to a certain partition group, if the version number of the data partition is not lower than the parameter value of the second version parameter of the partition group, the data partition is allowed to be transferred to the partition group, and when the parameter value of the first version parameter of the partition group is raised to the second version number, the parameter value is allowed to be rolled back to the original first version number. If the version number of the data partition is lower than the parameter value of the second version parameter of the partition group, the data partition is not permitted to be transferred temporarily until the data partition finishes data combination, namely, the version number of the data partition is increased to the second version number.
Specifically, it is assumed that the plurality of partition groups include a first partition group and a second partition group, the first partition group includes a first data partition for which data merging has not been performed, and a transfer task of transferring the first data partition to the second partition group is associated. If the version number of the first data partition is not lower than the parameter value of the second version parameter of the second partition group, for example, the version number of the first data partition and the parameter value of the second parameter of the second partition group are both the first version number, the transfer task may be performed to transfer the first data partition from the first partition group to the second partition group. Then, if the parameter value of the first version parameter of the second partition group is the second version number, the parameter value of the first version parameter may be rolled back to the first version number. In addition, if the version number of the first data partition is lower than the parameter value of the second version parameter of the second partition group, the execution of the transfer task may be suspended until the first data partition completes the data merge.
Next, the data partition transfer process will be described with reference to fig. 6, fig. 7, and fig. 8.
Referring to FIG. 6, a diagram of a data partition migration process is shown. Among them, the parameter name merge _ version shown in fig. 6 may be a parameter name of the third version parameter of the data partition. As shown in fig. 6, the partition group 1 includes the data partition 3 that has not been subjected to data merging, and is associated with a migration task of migrating the data partition 3 to the partition group 2. Based on the parameter value of merge _ version of the data partition 3, it can be known that the version number of the data partition 3 is 4. By comparing the version number 4 with the parameter value 4 of the determined _ merge _ version of the partition group 2, it can be known that the version number of the data partition 3 is not lower than the parameter value of the determined _ merge _ version of the partition group 2, so that the migration task can be performed to migrate the data partition 3 from the partition group 1 to the partition group 2.
After transferring data partition 3 from partition group 1 to partition group 2, the parameter value of current _ merge _ version of partition group 2 needs to be rolled back from 5 to 4 since the version number of data partition 3 has not yet been raised to 5. The display effect of the partition group 2 can then be shown in the right part of fig. 7. Fig. 7 is a schematic diagram illustrating the effect of partition group 1 and partition group 2.
In addition, since the data partition 3 has been transferred, and the parameter value of the merge _ version of each of the remaining data partitions in the partition group 1 is 5, it can be considered that the data merging is completed for each of the remaining data partitions in the partition group 1, and thus the parameter value of the current merge _ version of the partition group 1 can be updated from 4 to 5. At this time, the exhibition effect of the partition group 1 can be as shown in the left part of fig. 7.
In one example, assuming that the plurality of partition groups only include partition group 1 and partition group 2, whether the parameter value of current _ merge _ version of each partition group is 5 is determined by sequentially polling partition group 2 and partition group 1. If partition group 2 is polled before transferring data partition 3 to partition group 2, then parameter value 5 for current _ merge _ version of partition group 2 is obtained. If polling partition group 1 occurs when data partition 3 is transferred to partition group 2 and the parameter value for current _ merge _ version for partition group 1 has been updated from 4 to 5, then parameter value 5 for current _ merge _ version for partition group 1 is obtained. At this time, based on the obtained two parameter values 5, it is considered that the data partitions of the partition group 1 and the partition group 2 both complete data merging, so as to update the parameter value of the determined _ merge _ version of the partition group 1 and the partition group 2.
Since the parameter value of current _ merge _ version of partition group 1 is 5, the parameter value of determined _ merge _ version of partition group 1 can be updated from 4 to 5. Since the parameter value for current _ merge _ version of partition group 2 has rolled back to 4, the parameter value for determined _ merge _ version of partition group 2 is not modified, which is still 4.
After the parameter value of current _ merge _ version of partition group 2 is backed off from 5 to 4 and the parameter value of determined _ merge _ version of partition group 1 is updated from 4 to 5, if the target database yields the transfer task of transferring data partition 3 from partition group 2 to partition group 1, since the version number 4 of data partition 3 is lower than the parameter value 5 of determined _ merge _ version of partition group 1, the transfer of data partition 3, i.e., the execution of the transfer task, is suspended as shown in fig. 8 until data merge of data partition 3 is completed. FIG. 8 is a diagram illustrating a data partition migration process.
With further reference to fig. 9, the present specification provides an embodiment of a data merge processing apparatus for a target database, which corresponds to the method embodiment shown in fig. 2. The target database comprises a plurality of partition groups, wherein any partition group comprises a plurality of data partitions, and is provided with a first version parameter and a second version parameter.
As shown in fig. 9, the data merge processing apparatus 900 for a target database of the present embodiment includes: a first updating unit 901, a determining unit 902 and a second updating unit 903. The first updating unit 901 is configured to update a parameter value of a first version parameter of any target partition group in the plurality of partition groups from a current first version number to a second version number in response to detecting that each data partition of the target partition group completes data merging; wherein the second version number is higher than the first version number; the determining unit 902 is configured to determine whether parameter values of the first version parameters of the respective plurality of partition groups are all the second version numbers; the second updating unit 903 is configured to update the parameter value of the second version parameter of each of the plurality of partition groups from the first version number to the second version number if the determination result of the determining unit 902 is yes.
In some embodiments, the plurality of partition groups include a first partition group and a second partition group, the first partition group includes a first data partition that has not been subjected to data merging, and is associated with a transfer task of transferring the first data partition to the second partition group; and the apparatus 900 may further include: a partition transfer unit (not shown in the figure) configured to perform a transfer task to transfer the first data partition from the first partition group to the second partition group if the version number of the first data partition is not lower than the parameter value of the second version parameter of the second partition group.
In some embodiments, the first updating unit 901 may be further configured to: after the partition transfer unit transfers the first data partition from the first partition group to the second partition group, if the parameter value of the first version parameter of the second partition group is the second version number, the parameter value of the first version parameter is rolled back to the first version number.
In some embodiments, the partition transfer unit may be further configured to: and if the version number of the first data partition is lower than the parameter value of the second version parameter of the second partition group, suspending the execution of the transfer task until the first data partition completes data merging.
In some embodiments, each data partition of any partition group is provided with a third version parameter; and the first updating unit 901 may be further configured to: if the fact that any data partition in the target partition group completes data combination is detected, updating a parameter value of a third version parameter of the any data partition from a first version number to a second version number; and when the parameter values of the third version parameters of the data partitions of the target partition group are the second version numbers, determining that the data partitions of the target partition group complete data merging.
In some embodiments, the apparatus 900 may further include: a merging initiator (not shown in the figure) configured to. Respectively sending a data merging request to the plurality of partition groups, wherein the data merging request comprises a second version number; so that the partition groups merge the data in the respective data partitions.
In some embodiments, the determining unit 902 may be further configured to: sending parameter value acquisition requests aiming at the first version parameters to the plurality of partition groups respectively, receiving acquisition results returned by the plurality of partition groups respectively, and determining whether the parameter values of the first version parameters of the plurality of partition groups are all the second version numbers according to the received acquisition results.
In some embodiments, the second updating unit 903 may be further configured to: and sending parameter value updating requests aiming at the second version parameters to the plurality of partition groups respectively so that the plurality of partition groups respectively update the parameter values of the second version parameters from the first version numbers to the second version numbers.
In some embodiments, the second updating unit 903 may be further configured to: receiving the updating results respectively returned by the plurality of partition groups; when the received updating results are all updating success, determining that the combination is completed; when at least one of the update results is an update failure, after a time length is set at intervals, the instruction determining unit 902 continues to determine whether the parameter values of the first version parameters of the plurality of partition groups are all the second version numbers; and the updating result belonging to the updating failure is generated by the corresponding partition group in response to the parameter value of the first version parameter returning to the first version number.
In some embodiments, the apparatus 900 may include a merge controller, and the first updating unit 901, the determining unit 902, the second updating unit 903, and the merge initiating unit as described above may be specifically included in the merge controller.
In the embodiment of the apparatus corresponding to fig. 9, the detailed processing of each unit and the technical effect thereof can refer to the related description of the method embodiment in the foregoing, and are not repeated herein.
Embodiments of the present specification further provide a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed in a computer, the computer program is made to execute the data merging processing method for a target database, which is respectively described in the above method embodiments.
The embodiment of the present specification further provides a computing device, which includes a memory and a processor, where the memory stores executable code, and when the processor executes the executable code, the data merging processing method for a target database, which is respectively described in the foregoing method embodiments, is implemented.
Embodiments of the present specification further provide a computer program, where the computer program, when executed in a computer, causes the computer to execute the data merging processing method for the target database described in the above method embodiments respectively.
Those skilled in the art will recognize that the functionality described in the various embodiments disclosed herein may be implemented in hardware, software, firmware, or any combination thereof, in one or more of the examples described above. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium.
In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The above-mentioned embodiments, objects, technical solutions and advantages of the embodiments disclosed in the present specification are further described in detail, it should be understood that the above-mentioned embodiments are only specific embodiments of the embodiments disclosed in the present specification, and are not intended to limit the scope of the embodiments disclosed in the present specification, and any modifications, equivalent substitutions, improvements and the like made on the basis of the technical solutions of the embodiments disclosed in the present specification should be included in the scope of the embodiments disclosed in the present specification.
Claims (13)
1. A data merging processing method for a target database, wherein the target database comprises a plurality of partition groups, any partition group comprises a plurality of data partitions, and is provided with a first version parameter and a second version parameter, and the method comprises the following steps:
in response to the detection that the data merging of each data partition of any target partition group in the plurality of partition groups is completed, updating the parameter value of the first version parameter of the target partition group from the current first version number to a second version number; wherein the second version number is higher than the first version number;
determining whether parameter values of respective first version parameters of the plurality of partition groups are all the second version numbers;
if the determination result is yes, updating the parameter value of the second version parameter of each of the plurality of partition groups from the first version number to the second version number.
2. The method of claim 1, wherein the plurality of partition groups comprises a first partition group and a second partition group, the first partition group comprising a first data partition that has not been data merged and having associated therewith a transfer task to transfer the first data partition to the second partition group; and
the method further comprises the following steps:
if the version number of the first data partition is not lower than the parameter value of the second version parameter of the second partition group, executing the transfer task to transfer the first data partition from the first partition group to the second partition group.
3. The method of claim 2, wherein after transferring the first data partition from the first partition group to the second partition group, further comprising:
and if the parameter value of the first version parameter of the second partition group is the second version number, returning the parameter value of the first version parameter to the first version number.
4. The method of claim 2 or 3, further comprising:
and if the version number of the first data partition is lower than the parameter value of the second version parameter of the second partition group, suspending the execution of the transfer task until the first data partition completes data merging.
5. The method of claim 1, wherein each data partition of any partition group is provided with a third version parameter; and
the detecting that each data partition of any target partition group in the plurality of partition groups completes data merging specifically includes:
if it is detected that any data partition in the target partition group completes data merging, updating a parameter value of a third version parameter of the any data partition from the first version number to the second version number;
and when the parameter values of the third version parameters of the data partitions of the target partition group are the second version numbers, determining that the data partitions of the target partition group complete data merging.
6. The method of claim 1, wherein the target database comprises a merge controller.
7. The method of claim 6, further comprising:
sending, via the merge controller, a data merge request to the plurality of partition groups, respectively, the data merge request including the second version number; such that the plurality of partition groups merge data within respective data partitions.
8. The method of claim 6, wherein the determining whether the parameter values of the first version parameter of each of the plurality of partition groups are all the second version number comprises:
and respectively sending parameter value acquisition requests aiming at the first version parameters to the plurality of partition groups through the merging controller, receiving acquisition results respectively returned by the plurality of partition groups, and determining whether the parameter values of the first version parameters of the plurality of partition groups are the second version numbers according to the received acquisition results.
9. The method of claim 6, wherein said updating parameter values for a second version parameter for each of the plurality of partition groups from the first version number to the second version number comprises:
sending, via the merge controller, a parameter value update request for a second version parameter to the plurality of partition groups, respectively, so that the plurality of partition groups each update a parameter value of their second version parameter from the first version number to the second version number.
10. The method of claim 9, after sending parameter value update requests for the second version parameters to the plurality of partition groups, respectively, further comprising:
receiving the updating results respectively returned by the plurality of subarea groups;
when the received updating results are all updating success, determining that the combination is completed;
when at least one of the updating results is updating failure, after the interval is set for a long time, continuing to execute the step of determining whether the parameter values of the first version parameters of the plurality of partition groups are the second version numbers; and the updating result belonging to the updating failure is generated by the corresponding partition group in response to the parameter value of the first version parameter returning to the first version number.
11. A data merging processing device for a target database, wherein the target database comprises a plurality of partition groups, any partition group comprises a plurality of data partitions, and is provided with a first version parameter and a second version parameter, and the device comprises:
a first updating unit configured to update a parameter value of a first version parameter of any target partition group of the plurality of partition groups from a current first version number to a second version number in response to detecting that each data partition of the target partition group completes data merging; wherein the second version number is higher than the first version number;
a determining unit configured to determine whether parameter values of the first version parameters of each of the plurality of partition groups are all the second version numbers;
a second updating unit configured to update a parameter value of a second version parameter of each of the plurality of partition groups from the first version number to the second version number if a result of the determination by the determining unit is yes.
12. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed in a computer, causes the computer to carry out the method of any one of claims 1-10.
13. A computing device comprising a memory and a processor, wherein the memory has stored therein executable code that when executed by the processor implements the method of any of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210947846.0A CN115033585B (en) | 2022-08-09 | 2022-08-09 | Data merging processing method and device for target database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210947846.0A CN115033585B (en) | 2022-08-09 | 2022-08-09 | Data merging processing method and device for target database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115033585A CN115033585A (en) | 2022-09-09 |
| CN115033585B true CN115033585B (en) | 2022-11-15 |
Family
ID=83130180
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210947846.0A Active CN115033585B (en) | 2022-08-09 | 2022-08-09 | Data merging processing method and device for target database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115033585B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112306995A (en) * | 2020-11-11 | 2021-02-02 | 支付宝(杭州)信息技术有限公司 | Method and device for realizing database operation |
| CN112817995A (en) * | 2021-02-22 | 2021-05-18 | 北京奇艺世纪科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6343299B1 (en) * | 1998-11-16 | 2002-01-29 | International Business Machines Corporation | Method and apparatus for random update synchronization among multiple computing devices |
| CN103294675B (en) * | 2012-02-23 | 2018-08-03 | 上海盛大网络发展有限公司 | Data-updating method and device in a kind of distributed memory system |
| US20150120663A1 (en) * | 2012-06-01 | 2015-04-30 | Thomson Licensing | Method of data storing and data synchronization in a distributed data storage system |
| CN113761052A (en) * | 2020-11-27 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Database synchronization method and device |
-
2022
- 2022-08-09 CN CN202210947846.0A patent/CN115033585B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112306995A (en) * | 2020-11-11 | 2021-02-02 | 支付宝(杭州)信息技术有限公司 | Method and device for realizing database operation |
| CN112817995A (en) * | 2021-02-22 | 2021-05-18 | 北京奇艺世纪科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115033585A (en) | 2022-09-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9020901B2 (en) | Balancing database workloads through migration | |
| US20190377821A1 (en) | System and method for persistence and replication of changes to a data store | |
| US10915311B2 (en) | Automated firmware update management on huge big-data clusters | |
| CN108183957B (en) | Master-slave synchronization method and device | |
| CN112083889A (en) | Data migration method, device, equipment and readable storage medium | |
| US20120030513A1 (en) | Mechanism to Provide Assured Recovery for Distributed Application | |
| CN109063005B (en) | Data migration method and system, storage medium and electronic device | |
| US20200133657A1 (en) | Method, electronic device and computer readable storage medium of storage management | |
| CN108776643B (en) | Target code merging control method and system based on version control process | |
| JP2010092464A (en) | Method, system and computer program, for performing two-way orphan reconciliation in hierarchical storage management (hsm) control storage environment | |
| US20080114962A1 (en) | Silent memory reclamation | |
| CN113742135A (en) | Data backup method and device and computer readable storage medium | |
| CN112434008A (en) | Distributed database upgrading method, device and medium | |
| WO2021238798A1 (en) | Data storage method, apparatus, and computer system | |
| CN115033585B (en) | Data merging processing method and device for target database | |
| CN107168774B (en) | Virtual machine migration method and system based on local storage | |
| CN111147541B (en) | Node processing method, device and equipment based on parameter server and storage medium | |
| CN111373369A (en) | Software build using cloud system | |
| CN103136043A (en) | Method, system and network node for transferring asynchronous input and output (IO) | |
| JPH1091405A (en) | Software maintenance method | |
| US20220382635A1 (en) | Method and apparatus for processing transaction | |
| CN117177050B (en) | Virtual machine VPC network configuration snapshot processing method and system | |
| CN110874320B (en) | Data updating method, prefire server and readable storage medium | |
| CN116991374B (en) | Control method, device, electronic equipment and medium for constructing continuous integration task | |
| US10776148B1 (en) | System and method for utilizing computational power of a server farm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |