TW201828119A - Data synchronizing method and device avoiding the situation that some Threads have completed the synchronizing mission, but the other Threads in the same mission are still at the waiting state - Google Patents

Abstract

The embodiment of the present invention provides a method to generate the synchronized Threads that are expected to operate based on the to-be-synchronized data amount in the source database and the priority grades of the synchronizing mission. Proceed Thread allocation according to the number of synchronized Threads that are expected to operate, and synchronize the allocated Threads for the said to-be-synchronized data to the target database. Based on the embodiment of the present invention, Threads can be dynamically allocated with respect to the practical situation of the synchronizing mission, avoiding the situation that some Threads have completed the synchronizing mission, but the other Threads in the same mission are still at the waiting state, thereby increasing the efficiency and stability of data synchronization.

Description

 Data synchronization method and device  

The present application relates to the field of Internet technologies, and in particular, to a data synchronization method and a data synchronization device.

A data synchronization task is a series of periodic scheduling tasks created for data synchronization between different databases. In a large data scheduling system, there are often a large number of data synchronization tasks running at the same time. For example, the data of HDFS (Hadoop Distributed File System) is synchronized to mySQL (relational database management system), or from HDFS synchronization data to Hbase (Hadoop Database, distributed storage system).

When using the data synchronization task to synchronize the data of the source database to the target database, the user usually needs to set a synchronization thread in advance, and the synchronization controller calls the corresponding thread according to the preset synchronization thread number and distributes it to a Or a plurality of synchronous processing devices, and the data processing is performed by the synchronous processing device. However, when the amount of data of the data to be synchronized of a synchronization task is large, if the thread of the synchronization task is distributed to the relatively busy synchronization processing device, it cannot be synchronized by using the preset synchronization thread number. Some threads have completed the synchronization task, while other threads of the same task are still waiting, resulting in a lower average synchronization speed for the entire synchronization task. Therefore, the current data synchronization method cannot effectively utilize the synchronization processing capability of the synchronous processing device, and there is a problem that the data synchronization efficiency is low.

In view of the above problems, embodiments of the present application have been made in order to provide a data synchronization method and a corresponding data synchronization apparatus that overcome the above problems or at least partially solve the above problems.

In order to solve the above problem, the present application discloses a data synchronization method, including: generating a synchronization execution thread that is expected to run according to the data amount of the data to be synchronized in the source database and the priority of the synchronization task; The number is configured by the thread, and the configured thread is used to synchronize the data to be synchronized to the target database.

Optionally, performing the thread configuration according to the synchronization execution thread that is expected to run includes: determining whether the synchronization execution number of the desired operation is greater than a synchronization execution number required for operation; if yes, operating according to the expected operation The difference between the number of synchronization threads and the number of synchronization threads required for the operation is supplemented by the corresponding thread.

Optionally, before the generating the number of synchronization threads to be run according to the data amount of the data to be synchronized and the priority of the synchronization task in the source database, the method further includes: from the data synchronization task submitted by the user, Obtaining a priority of the synchronization task, and acquiring the data amount of the to-be-synchronized data from the source database in which the data to be synchronized is saved.

Optionally, the generating, according to the data amount of the data to be synchronized in the source database and the priority of the synchronization task, generating the synchronization execution thread that is expected to run includes: searching for the data amount of the to-be-synchronized data and the priority of the synchronization task The average synchronization rate is calculated by using the average synchronization rate, the maximum synchronization rate of the synchronization processing device, and the number of threads that the synchronization processing device can use for synchronization processing.

Optionally, before the determining whether the expected number of synchronization threads to be run is greater than the number of synchronization threads required for operation, the method further includes: the target database from which data is to be synchronized with the source database. Get the number of synchronization threads required for the run.

Optionally, before the using the configured thread to synchronize the to-be-synchronized data to the target database, the method further includes: splitting the to-be-synchronized data according to the synchronization execution number required for the operation And a plurality of to-be-synchronized data blocks; wherein the configured thread is synchronized to the target database for the data to be synchronized: a synchronization thread for synchronizing each to-be-synchronized data block is scheduled to a synchronization processing device, The synchronization thread is processed by the synchronization processing device.

Optionally, the using the configured thread to synchronize the data to be synchronized to the target database comprises: sending at least one synchronization thread that meets a preset condition to the synchronization processing device to the synchronization processing device.

Optionally, the preset condition includes at least one of the following: the at least one synchronization thread belongs to the same data synchronization task, the to-be-processed time is greater than a preset time threshold, the priority of the corresponding synchronization task is greater than a preset priority threshold, A single synchronous processing device can be processed synchronously.

Optionally, the using the configured thread to synchronize the data to be synchronized to the target database comprises: sending the synchronization thread preferentially to a maximum number of threads available for synchronization processing that is greater than a preset number of threads threshold Synchronize the device.

In order to solve the above problem, the present application further discloses a data synchronization device, comprising: a synchronization execution number generation module, configured to generate a synchronization of a desired operation according to a data amount of a data to be synchronized in a source database and a priority of a synchronization task. The thread is executed; the synchronization processing module is configured to perform thread configuration according to the number of synchronization threads that are expected to be run, and synchronize the data to be synchronized to the target database by using the configured thread.

Optionally, the synchronization processing module includes: a synchronization execution number determination sub-module, configured to determine whether the synchronization execution number of the desired operation is greater than a synchronization execution number required for operation, and if yes, execute the execution The thread complementing the sub-module; the thread complementing the sub-module, configured to supplement the configuration of the corresponding thread according to the difference between the number of synchronization threads that are expected to run and the number of synchronization threads required for the operation.

Optionally, the device further includes: a priority and a data volume obtaining module, configured to acquire a priority of the synchronization task from a data synchronization task submitted by the user, and from a source database that stores the data to be synchronized Obtaining the amount of data of the data to be synchronized.

Optionally, the synchronization execution number generation module includes: an average synchronization rate search sub-module, configured to search for an average synchronization rate of the data amount of the to-be-synchronized data and the priority of the synchronization task; And a calculation submodule, configured to calculate the number of synchronization threads of the desired operation by using the average synchronization rate, a maximum synchronization rate of the synchronization processing device, and a number of threads that the synchronization processing device can use for synchronization processing.

Optionally, the device further includes: a running synchronization execution number acquisition module, configured to acquire the synchronization execution number required for the operation from the target data database to be synchronized with the source database.

Optionally, the device further includes: a data splitting module to be synchronized, configured to split the to-be-synchronized data into a plurality of to-be-synchronized data blocks according to the synchronization execution number required for the operation; The processing module is specifically configured to: schedule a synchronization thread for synchronizing each data block to be synchronized, and schedule the synchronization thread to the synchronization processing device to process the synchronization thread by the synchronization processing device.

Optionally, the synchronization processing module is specifically configured to: send the at least one synchronization thread that meets the preset condition to the synchronization processing device.

Optionally, the preset condition includes at least one of the following: the at least one synchronization thread belongs to the same data synchronization task, the to-be-processed time is greater than a preset time threshold, the priority of the corresponding synchronization task is greater than a preset priority threshold, A single synchronous processing device can be processed synchronously.

Optionally, the synchronization processing device includes multiple, the synchronization processing module is specifically configured to: send the synchronization thread preferentially to the synchronization that the maximum number of threads available for synchronization processing is greater than a preset threshold number threshold Processing equipment.

Embodiments of the present application include the following advantages:

According to the embodiment of the present application, the number of synchronization threads expected to run is generated according to the data amount of the data to be synchronized in the source database and the priority of the synchronization task, and the corresponding number of threads are configured according to the number of synchronization execution threads that are expected to be run. Therefore, the thread can be dynamically configured for the actual situation of the synchronization task, to avoid the situation that some threads have completed the synchronization task, and other threads of the same task are still in the waiting state, which improves the efficiency and stability of data synchronization. Moreover, the number of synchronization threads that are expected to run is dynamically adjusted according to the priority of the synchronization task, so that the synchronization tasks with higher importance can be processed preferentially.

301‧‧‧Synchronous thread generation module

302‧‧‧Synchronization Processing Module

401‧‧‧Priority and data acquisition module

402‧‧‧Synchronous thread acquisition module

403‧‧‧Synchronous thread generation module

404‧‧‧Synchronization Processing Module

1 is a flow chart of a first embodiment of a data synchronization method of the present application; FIG. 2 is a flow chart of a second embodiment of a data synchronization method of the present application; FIG. 3 is a first embodiment of a data synchronization device of the present application. Figure 4 is a block diagram showing the structure of a second embodiment of a data synchronizing apparatus of the present application.

The above described objects, features and advantages of the present application will become more apparent and understood.

Referring to FIG. 1 , a flow chart of the steps of the first embodiment of the data synchronization method of the present application is shown, which may specifically include the following steps:

Step 101: Generate, according to the data volume of the data to be synchronized in the source database and the priority of the synchronization task, the number of synchronization threads that are expected to run.

It should be noted that the number of synchronization threads that are expected to run may be the number of first-time synchronization threads or multiple rounds of synchronization threads preset by the user. Because the user cannot know the situation of the data to be synchronized and the running status of the synchronous processing device in this synchronization task, or the user does not know how to set it, the number of synchronization threads set may be the synchronization thread actually needed when running the synchronization. The number does not match, causing the synchronization thread to be distributed to the busy synchronous processing device during synchronization; or the user sets the number of synchronous threads that are expected to run to a fixed value, resulting in a synchronization task not being properly synchronized. Distribution affects synchronization efficiency. In addition, the important programs of the synchronization tasks of different data are different, and the synchronization controller treats each synchronization task equally, resulting in important synchronization threads failing to prioritize synchronization due to lack of processing resources. Therefore, using the current data synchronization method for data synchronization may lead to the above problems, which ultimately leads to low data synchronization efficiency and lack of stability in the processing of synchronization tasks.

In the specific implementation of the embodiment of the present application, the amount of data to be synchronized in the source database and the priority of the synchronization task may be utilized to generate a synchronization execution number of the desired operation. In an actual application, a user can submit a data synchronization task, which usually includes information about the priority of the synchronization task, so that the priority of the synchronization task can be obtained therefrom. For the data synchronization task submitted by the user, the amount of data of the synchronization data can be obtained from the source database of the data source to be synchronized. Of course, those skilled in the art may use other means to obtain information such as the amount of data and priority, such as embedding information of the data to be synchronized and the priority of the synchronization task in the data synchronization task, so as to synchronize from the submitted data. Get the amount of data and priority directly in the task.

There are many ways to generate the number of synchronous threads that you want to run based on the amount of data and priority. For example, a data amount-priority-average synchronization rate comparison table may be preset, and a combination of different data amounts and priorities in the comparison table corresponds to a specific average synchronization rate. Find the corresponding average synchronization rate based on the amount of data and priority. The average synchronization rate to be searched, at the synchronization rate that each CPU (Central Processing Unit) on the synchronous processing device can run, can determine the number of CPUs required for synchronization, and can execute the number of CPUs as synchronous execution of the desired operation. Number of threads.

Step 102: Perform thread configuration according to the number of synchronization execution threads that are expected to be run, and synchronize the data to be synchronized to the target database by using the configured thread.

The synchronization thread can be configured according to the number of synchronization threads that you want to run. There are various ways to configure the configuration. For example, you can configure the corresponding virtual thread according to the difference between the number of synchronous threads that you want to run and the number of synchronous threads that you need to run synchronously. Part of the CPU is placed in the standby state and pauses to process other synchronization tasks to ensure that the synchronization processing device has sufficient resources to process the data synchronization task; when the number of synchronization threads expected to run is the same as the number of synchronization threads actually required for the synchronous operation, You can configure the corresponding entity thread directly according to the number of synchronization threads you want to run.

The configured entity thread and/or virtual thread synchronization can be used to process the data to be synchronized. In an actual application, the data to be synchronized may be first split into a plurality of data blocks to be synchronized, and each configured thread is used to synchronize a plurality of data blocks to be synchronized. According to the remaining processing resources of the synchronization processing device, the plurality of threads are scheduled to one or more synchronization processing devices, and the plurality of to-be-synchronized data blocks are synchronized by the synchronization processing device to the target data library.

When the synchronization controller schedules the synchronization thread to the synchronization processing device, it can be determined whether the number of synchronous execution threads actually required for the synchronous operation is greater than the number of synchronous execution threads that are expected to be run, and the scheduling is performed according to the determination result.

For example, when the number of synchronization threads actually required for synchronous operation is greater than the number of synchronization threads expected to run, the synchronization controller may adopt a data synchronization mode of multi-machine multi-thread to prepare a first round entity thread configured for the synchronization task. The scheduling of the thread that has the longest waiting time and the highest priority and belongs to the same synchronization task is sent to multiple synchronization processing devices to ensure that the threads of the synchronization task can be preferentially processed to avoid a certain thread having been completed. Synchronization, while other threads that belong to the same synchronization task are still waiting for processing, affecting the average synchronization rate of the synchronization task.

When the number of synchronous threads required for synchronous operation is smaller than the number of synchronous threads that are expected to run, the synchronous controller can use the data synchronization mode of the single-machine multi-thread to prepare the physical thread and the virtual thread row configured for the synchronization task. The thread that has the longest waiting time, the highest priority, and belongs to the same synchronous task, is preferentially sent to a synchronous processing device that can support the number of synchronous threads that are expected to run; if there is no synchronous processing device that satisfies the condition , a synchronization processing device with the most remaining processing resources can be preferentially selected as the transmission target.

Certainly, those skilled in the art can use the data synchronization mode of a single machine single thread and a single machine single thread to perform scheduling according to actual conditions, and the load balancing schedule can be performed according to the processing resources of the synchronous processing device during scheduling, or can be randomized. The scheduling is not limited in this embodiment of the present application.

According to the embodiment of the present application, the number of synchronization threads expected to run is generated according to the data amount of the data to be synchronized in the source database and the priority of the synchronization task, and the corresponding number of threads are configured according to the number of synchronization execution threads that are expected to be run. Therefore, the thread can be dynamically configured for the actual situation of the synchronization task, to avoid the situation that some threads have completed the synchronization task, and other threads of the same task are still in the waiting state, which improves the efficiency and stability of data synchronization. Moreover, the number of synchronization threads that are expected to run is dynamically adjusted according to the priority of the synchronization task, so that the synchronization tasks with higher importance can be processed preferentially.

Referring to FIG. 2, a flow chart of the steps of the second embodiment of the data synchronization method of the present application is shown. Specifically, the method may include the following steps:

In step 201, the priority of the synchronization task is obtained from the data synchronization task submitted by the user, and the data amount of the to-be-synchronized data is obtained from the source database in which the data to be synchronized is saved.

In a specific implementation, when data synchronization is required, the user can create a data synchronization task on the client and submit it to the corresponding synchronization processing server. When creating a data synchronization task, the client can generate a corresponding priority according to the importance of the data to be synchronized, and is included in the data synchronization task, so the synchronization processing server can obtain the priority of the synchronization task from the submitted data synchronization task. . In addition, the amount of data of the data to be synchronized can also be read from the post data of the source database in which the data to be synchronized is stored. For example, when HDFS is synchronized to Hbase, the amount of data of the data to be synchronized is obtained by reading the information of the HDFS, and other types of databases can also obtain the amount of data in a similar manner.

Step 202: Find an average synchronization rate for the data amount of the to-be-synchronized data and the priority of the synchronization task.

Step 203: Calculate the number of synchronization threads expected to run by using the average synchronization rate, the maximum synchronization rate of the synchronization processing device, and the number of threads that the synchronization processing device can use for synchronization processing.

The average synchronization rate of the synchronization task can be found according to the acquired data amount of the data to be synchronized and the priority of the synchronization task. Specifically, the expected average synchronization rate of the synchronization task may be initialized in advance according to the amount of data and the priority, for example:

According to the data volume of the data to be synchronized and the priority of the synchronization task in the current synchronization task, for example, a synchronization task with a data volume of 0-10G and a priority of 8 can find that the corresponding average synchronization task is 30 mb/ s.

Then, according to the processing resources of each synchronous processing device, the number of CPUs required for synchronization is calculated. For example, each synchronous processing device has 6 CPU cores with a maximum synchronization rate of 60 mb/s, and it can be calculated that the maximum synchronization rate that each CPU can run is 10 mb/s.

Dividing the expected average synchronization rate of the found synchronization task by 30 mb/s by the maximum synchronization rate that each CPU can run is 10 mb/s, and the number of CPUs required to be calculated is 3, that is, the number of synchronization threads expected to run. .

Step 204: Obtain the number of synchronization threads required for the operation from the target database to be synchronized with the source database.

The number of synchronization threads required for the operation can be read from the source data of the target database. For example, when the HDFS is synchronized to HBase, the number of Regions of the HBase target table can be read, and the number of Regions of the Hbase target table is required for operation. Synchronous thread count. Region is the basic unit of HBase data storage and management. Usually, Region determines the number of threads used by the target database during synchronization.

Step 205: Determine whether the number of synchronization executions of the desired operation is greater than the number of synchronization execution threads required for operation; if yes, proceed to step 206.

Step 206: Supplemental configuration of the corresponding thread according to the difference between the expected number of synchronization executions and the synchronization execution number required for the operation.

When the number of synchronous threads expected to run is greater than the number of synchronous threads actually required for synchronous operation, the difference between the number of synchronous threads required to run and the number of synchronized threads required for running may be subtracted, and the corresponding virtual thread may be supplemented. The configured virtual thread is used to put some CPUs on the synchronous processing device into a standby state and suspend processing other synchronization tasks to ensure that the synchronous processing device has sufficient resources to process the data synchronization task.

Step 207: Split the data to be synchronized into a plurality of data blocks to be synchronized according to the synchronization execution number required for the operation.

Step 208: Synchronize the data to be synchronized to the target database by using the configured thread.

As a preferred example of the embodiment of the present application, the step 208 may be specifically:

The synchronization thread for synchronizing the respective data blocks to be synchronized is scheduled to the synchronization processing device to process the synchronization thread by the synchronization processing device.

The data to be synchronized can be split into a plurality of data blocks to be synchronized, and each configured thread is used to synchronize a plurality of data blocks to be synchronized. According to the remaining processing resources of the synchronization processing device, the plurality of threads are scheduled to one or more synchronization processing devices, and the plurality of to-be-synchronized data blocks are synchronized by the synchronization processing device to the target data library.

In an actual application, multiple threads corresponding to the data synchronization task may be submitted to the synchronization controller, and the synchronous controller performs the judgment process of whether the number of synchronous execution threads required to be run is greater than the number of synchronous execution threads required for the operation, and According to the judgment result, the configuration thread is supplemented correspondingly and the thread is scheduled to a plurality of synchronization processing devices.

As a preferred example of the embodiment of the present application, the step 208 may include:

Transmitting, by the synchronization processing device, the at least one synchronization thread that meets the preset condition to the synchronization processing device; wherein the preset condition may include at least one of the following:

The at least one synchronization thread belongs to the same data synchronization task, the to-be-processed time is greater than the preset time threshold, the priority of the corresponding synchronization task is greater than the preset priority threshold, and the single synchronization processing device can be processed synchronously.

For the case where the number of synchronous threads expected to run is greater than the number of synchronous threads actually required for the synchronous operation, the synchronous controller may adopt the data synchronization mode of the single-machine multi-thread to prepare the entity thread and the virtual thread configured for the synchronization task. The schedule of the processing of the same data synchronization task whose priority is greater than the preset time threshold and whose priority is greater than the preset priority threshold is preferentially sent to the synchronization processing device that can be synchronously processed by a single synchronization processing device; The synchronization processing device that does not satisfy the condition may preferentially select one synchronization processing device with the largest remaining processing resource as the transmission target.

As a preferred example of the embodiment of the present application, the synchronization processing device includes multiple, and the step 208 may include: sending the synchronization thread preferentially to the maximum number of threads available for synchronization processing is greater than a preset thread threshold. Synchronous processing device.

In actual applications, there may be no synchronization processing device that can satisfy the preset condition that can be synchronously processed by a single synchronization processing device. In this case, the synchronization thread can be preferentially sent to the maximum number of threads available for synchronization processing, which is greater than the preset thread. On the plurality of synchronization processing devices of the threshold, that is, the synchronization processing device that selects more CPU processing resources performs the first or more rounds of data synchronization. For the scheduling of the multiple synchronous processing devices, the load balancing scheduling may be performed according to the processing resources of the synchronous processing device, or may be randomly scheduled, which is not limited in this embodiment of the present application.

According to the embodiment of the present application, the corresponding virtual thread is supplemented according to the difference between the number of synchronization threads that are expected to run and the number of synchronization threads required for the operation, and some CPUs on the synchronous processing device are placed in the standby state by using the virtual thread. The other synchronization tasks are suspended to ensure that the synchronization processing device has sufficient resources to process the data synchronization task, thereby improving the efficiency and stability of data synchronization. Moreover, there is no need to control the use of the CPU from the system level, which increases the flexibility of data synchronization.

It should be noted that, for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the embodiments of the present application are not limited by the described action sequence, because In accordance with embodiments of the present application, certain steps may be performed in other sequences or concurrently. In the following, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required in the embodiments of the present application.

Referring to FIG. 3, a block diagram of a first embodiment of a data synchronizing apparatus of the present application is shown. Specifically, the following modules may be included:

The synchronization execution number generation module 301 is configured to generate a synchronization execution thread number to be operated according to the data amount of the data to be synchronized and the priority of the synchronization task in the source database.

The synchronization processing module 302 is configured to perform thread configuration according to the number of synchronization execution threads that are expected to be run, and synchronize the data to be synchronized to the target data library by using the configured thread.

According to the embodiment of the present application, the number of synchronization threads expected to run is generated according to the data amount of the data to be synchronized in the source database and the priority of the synchronization task, and the corresponding number of threads are configured according to the number of synchronization execution threads that are expected to be run. Therefore, the thread can be dynamically configured for the actual situation of the synchronization task, to avoid the situation that some threads have completed the synchronization task, and other threads of the same task are still in the waiting state, which improves the efficiency and stability of data synchronization. Moreover, the number of synchronization threads expected to run is dynamically adjusted according to the priority of the synchronization task, so that the more important synchronization tasks can be processed preferentially.

Referring to FIG. 4, a block diagram of a second embodiment of a data synchronization device of the present application is shown. Specifically, the following modules may be included:

The priority and data volume obtaining module 401 is configured to obtain the priority of the synchronization task from the data synchronization task submitted by the user, and obtain the data of the to-be-synchronized data from the source database that stores the data to be synchronized. the amount.

The running synchronization execution number acquisition module 402 is configured to obtain the synchronization execution number required for the operation from the target data base to be synchronized with the source database.

The synchronization execution number generation module 403 is configured to generate a synchronization execution thread number to be operated according to the data amount of the data to be synchronized in the source database and the priority of the synchronization task.

The synchronization processing module 404 is configured to perform thread configuration according to the number of synchronization threads that are expected to be run, and synchronize the data to be synchronized to the target database by using the configured thread.

As a preferred example of the embodiment of the present application, the synchronization execution number generation module 403 may include: an average synchronization rate search sub-module, configured to search for an average of the data amount of the to-be-synchronized data and the priority of the synchronization task. a synchronization rate calculation sub-module, configured to calculate the synchronization of the desired operation by using the average synchronization rate, a maximum synchronization rate of the synchronization processing device, and a number of threads that the synchronization processing device can process synchronously The number of threads.

As a preferred example of the embodiment of the present application, the synchronization processing module 404 may include: a synchronization execution number determination sub-module, configured to determine whether the expected number of synchronization execution threads is greater than the number of synchronization execution threads required for operation If it is, it is called using the thread to supplement the sub-module.

The thread complementing sub-module is configured to supplement the configuration of the corresponding thread according to the difference between the number of synchronization threads that are expected to run and the number of synchronization threads required for the operation.

As a preferred example of the embodiment of the present application, the device may further include: a data splitting module to be synchronized, configured to split the data to be synchronized into multiple to be synchronized according to the number of synchronization threads required for the operation Data block.

As a preferred example 1 of the embodiment of the present application, the synchronization processing module 404 may be specifically configured to: schedule a synchronization thread for synchronizing each to-be-synchronized data block to a synchronization processing device, to be used by the synchronization processing device Processing the synchronization thread.

As a preferred example 2 of the embodiment of the present application, the synchronization processing module 404 may be specifically configured to: send the at least one synchronization thread that meets the preset condition to the synchronization processing device.

As a preferred example of the embodiment of the present application, the preset condition includes at least one of the following:

The at least one synchronization thread belongs to the same data synchronization task, the to-be-processed time is greater than the preset time threshold, the priority of the corresponding synchronization task is greater than the preset priority threshold, and the single synchronization processing device can be processed synchronously.

As a preferred example 3 of the embodiment of the present application, the synchronization processing device includes a plurality, and the synchronization processing module 404 may be specifically configured to: send the synchronization thread preferentially to a maximum number of threads available for synchronization processing, which is greater than A synchronization processing device that presets the threshold number of threads.

According to the embodiment of the present application, the corresponding virtual thread is supplemented according to the difference between the number of synchronization threads that are expected to run and the number of synchronization threads required for the operation, and some CPUs on the synchronous processing device are placed in the standby state by using the virtual thread. The other synchronization tasks are suspended to ensure that the synchronization processing device has sufficient resources to process the data synchronization task, thereby improving the efficiency and stability of data synchronization. Moreover, there is no need to control the use of the CPU from the system level, which increases the flexibility of data synchronization.

For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The various embodiments in the present specification are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same similar parts between the various embodiments can be referred to each other.

Those skilled in the art will appreciate that embodiments of the embodiments of the present application can be provided as a method, apparatus, or computer program product. Thus, embodiments of the present application may take the form of a complete hardware embodiment, a fully software embodiment, or an embodiment combining soft and hardware aspects. Moreover, embodiments of the present application may employ computer program products embodied on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable code therein. form.

In a typical configuration, the computer device includes one or more processors (CPUs), input/output interfaces, a network interface, and memory. Memory memory may include non-persistent memory, random access memory (RAM) and/or non-volatile memory memory in computer readable media, such as read only memory (ROM) or flash memory. Body (flash RAM). Memory memory is an example of a computer readable medium. Computer readable media including both permanent and non-permanent, removable and non-removable media can be stored by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other materials. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), and other types of random access memory (RAM). ), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only optical disk read-only memory (CD-ROM), digital versatile CD-ROM (DVD) or other optical storage, magnetic tape cartridge, magnetic tape storage or other magnetic storage device or any other non-transportable media that can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include non-persistent computer readable media, such as modulated data signals and carrier waves.

The embodiments of the present application are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowcharts and/or block diagrams, and combinations of flows and/or blocks in the flowcharts and/or block diagrams can be implemented by computer program instructions. The computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing terminal device to generate a machine such that instructions are executed by a processor of a computer or other programmable data processing terminal device Means are provided for implementing the functions specified in one or more blocks of a flowchart or a plurality of processes and/or square blocks.

The computer program instructions can also be stored in a computer readable storage that can direct a computer or other programmable data processing terminal device to operate in a particular manner such that instructions stored in the computer readable storage produce an article of manufacture including the instruction device. The instruction device implements the functions specified in one or more blocks of a flowchart or a plurality of flows and/or square blocks of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device such that a series of operational steps are performed on a computer or other programmable terminal device to produce computer implemented processing for use in a computer or other programmable terminal device The instructions executed on the steps provide steps for implementing the functions specified in one or more blocks of a flowchart or a plurality of flows and/or square blocks of the flowchart.

While a preferred embodiment of the embodiments of the present application has been described, those skilled in the art can make further changes and modifications to the embodiments once they are aware of the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and the modifications and

Finally, it should also be noted that in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities. There is any such actual relationship or order between operations. Moreover, the terms "comprise," "comprise," or "include" or "include" or "include" or "include" or "includes" Other requirements are also included in the process, method, article or terminal equipment. In the absence of further limitations, the requirement defined by the phrase "comprising a ..." does not exclude the presence of additional equivalents in the process, method, article, or terminal device that includes the claim.

The above is a detailed description of a data synchronization method and a data synchronization device provided by the present application. The principles and implementation manners of the present application are described in the specific examples. The description of the above embodiments is only for helping to understand the present application. The method of application and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be understood. To limit the application.

Claims (18)

 A data synchronization method includes: generating, according to a data amount of a data to be synchronized in a source database and a priority of a synchronization task, a number of synchronous execution threads to be executed; performing thread configuration according to a desired number of synchronous execution threads And use the configured thread to synchronize the data to be synchronized to the target database.    The method of claim 1, wherein the performing thread configuration according to the expected number of synchronized execution threads comprises: determining whether the number of synchronization executions of the desired operation is greater than a number of synchronization execution threads required for operation; Then, according to the difference between the number of synchronous execution threads that are expected to run and the number of synchronous execution threads required for the operation, the corresponding thread is additionally configured.    The method of claim 1, wherein the method further comprises: before generating the number of synchronization threads to be run according to the amount of data to be synchronized in the source database and the priority of the synchronization task. In the data synchronization task submitted by the user, the priority of the synchronization task is obtained, and the data volume of the data to be synchronized is obtained from the source database in which the data to be synchronized is saved.    The method of claim 1, wherein the generating the synchronization thread to be run according to the data amount of the data to be synchronized in the source database and the priority of the synchronization task comprises: searching for the data to be synchronized The average synchronization rate of the data volume and the priority of the synchronization task; using the average synchronization rate, the maximum synchronization rate of the synchronization processing device, and the number of threads that the synchronization processing device can use for synchronization processing, calculate the number of synchronization threads that are expected to run.    The method of claim 2, wherein before the determining whether the expected number of synchronization executions is greater than the number of synchronization threads required for operation, the method further comprises: proceeding from the source database In the target database of data synchronization, obtain the number of synchronization threads required for the operation.    The method of claim 1, wherein before the thread configured to synchronize to the target database, the method further comprises: following the synchronization execution number required for the operation The data to be synchronized is split into a plurality of data blocks to be synchronized; the thread configured to synchronize the data to be synchronized to the target database is: a synchronization thread that is used to synchronize each data block to be synchronized, and scheduled to be synchronized. The device processes the synchronization thread by the synchronization processing device.    The method of claim 1, wherein the using the configured thread to synchronize the data to be synchronized to the target database comprises: sending at least one synchronization thread that satisfies a preset condition to the preset condition to the Synchronize the device.    According to the method of claim 7, the preset condition includes at least one of the following: the at least one synchronization thread belongs to the same data synchronization task, the to-be-processed time is greater than the preset time threshold, and the priority of the corresponding synchronization task is greater than The preset priority threshold and a single synchronous processing device can be processed synchronously.    According to the method of claim 1, wherein the synchronization processing device includes a plurality of, the configured thread for synchronizing the data to be synchronized to the target database includes: sending the synchronization thread to the priority The synchronous processing device whose maximum number of threads processed by synchronization is greater than the preset number of threads threshold.    A data synchronization device, comprising: a synchronization execution number generation module, configured to generate a synchronization execution number of a desired operation according to a data amount of a data to be synchronized in a source database and a priority of a synchronization task; The module is configured to perform thread configuration according to the number of synchronization threads that are expected to run, and uses the configured thread to synchronize the data to be synchronized to the target database.    The device of claim 10, wherein the synchronization processing module comprises: a synchronization execution number determination sub-module, configured to determine whether the synchronization execution number of the desired operation is greater than a synchronization execution thread required for operation The number, if it is, is called using the thread to supplement the sub-module; the thread complementing the sub-module is used to supplement the corresponding configuration according to the difference between the number of synchronization threads that are expected to run and the number of synchronization threads required for the operation. Thread.    The device of claim 10, wherein the device further comprises: a priority and a data volume acquisition module, configured to acquire a priority of the synchronization task from a data synchronization task submitted by the user, and save the In the source database of the data to be synchronized, the amount of data of the data to be synchronized is obtained.    The device according to claim 10, wherein the synchronization execution number generation module comprises: an average synchronization rate search sub-module, configured to search for a data amount of the to-be-synchronized data and a priority of the synchronization task. An average synchronization rate; a synchronization execution number calculation sub-module for calculating the synchronization thread of the desired operation by using the average synchronization rate, the maximum synchronization rate of the synchronization processing device, and the number of threads that the synchronization processing device can process synchronously number.    The device of claim 11, wherein the device further comprises: a running synchronization execution number acquisition module, configured to acquire the operation from a target database to be synchronized with the source database The number of synchronization threads required.    The device of claim 10, wherein the device further comprises: a data splitting module to be synchronized, configured to split the data to be synchronized into a plurality of to-be-scheduled according to the synchronization execution number required for the operation The synchronization processing module is configured to: synchronize the synchronization thread used for synchronizing the data blocks to be synchronized to the synchronization processing device, so that the synchronization processing device processes the synchronization thread.    The device of claim 10, wherein the synchronization processing module is configured to: preferentially send at least one synchronization thread that meets a preset condition to the synchronization processing device.    According to the device of claim 16, the preset condition includes at least one of the following: the at least one synchronization thread belongs to the same data synchronization task, the to-be-processed time is greater than the preset time threshold, and the priority of the corresponding synchronization task is greater than The preset priority threshold and a single synchronous processing device can be processed synchronously.    The device of claim 10, wherein the synchronization processing device comprises a plurality of, the synchronization processing module is specifically configured to: send the synchronization thread preferentially to the maximum number of threads available for synchronization processing is greater than Set the synchronization processing device of the thread number threshold.