CN106933534A - A kind of method of data synchronization and device - Google Patents
A kind of method of data synchronization and device Download PDFInfo
- Publication number
- CN106933534A CN106933534A CN201511032646.9A CN201511032646A CN106933534A CN 106933534 A CN106933534 A CN 106933534A CN 201511032646 A CN201511032646 A CN 201511032646A CN 106933534 A CN106933534 A CN 106933534A
- Authority
- CN
- China
- Prior art keywords
- thread
- data
- passes
- line synchro
- synchronized
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/30003—Arrangements for executing specific machine instructions
- G06F9/30076—Arrangements for executing specific machine instructions to perform miscellaneous control operations, e.g. NOP
- G06F9/30087—Synchronisation or serialisation instructions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The embodiment of the present application provides a kind of priority according to the data volume of data to be synchronized and synchronous task in source database, and the line synchro number of passes of operation is expected in generation；The line synchro number of passes for desirably running carries out thread configuration, and using the thread for configuring for the data syn-chronization to be synchronized to target database.According to the embodiment of the present application, the actual conditions dynamic configuration thread of synchronous task can be directed to, avoid the occurrence of part thread and complete synchronous task, and other threads of same task improve the efficiency and stability of data syn-chronization also in the situation of wait state.
Description
Technical field
The application is related to Internet technical field, more particularly to a kind of method of data synchronization and a kind of data
Sychronisation.
Background technology
A series of cycles that data syn-chronization task is directed to the data syn-chronization between disparate databases and creates
The task of scheduling.In a large-scale data dispatch system, often have a large amount of in the operation of same time
Data syn-chronization task.For example by HDFS (Hadoop Distributed File System, Hadoop point
Cloth file system) data syn-chronization to mySQL (Relational DBMS), Huo Zhecong
HDFS synchrodatas are to Hbase (Hadoop Database, distributed memory system).
When using data syn-chronization task by the data syn-chronization of source database to target database, user is usual
Need to pre-set a line synchro number of passes, isochronous controller is called accordingly according to default line synchro number of passes
Thread, be distributed to one or more synchronous processing equipments, data syn-chronization is carried out by synchronous processing equipment
Reason.However, when the data volume of the data to be synchronized of certain synchronous task is larger, if by the synchronous task
Thread be distributed to the busy synchronous processing equipment of comparing, it cannot call and meet default line synchro number of passes
Synchronize, possible part thread has completed synchronous task, and other threads of same task also in
Wait state, so as to cause the average synchronizing speed of whole synchronous task relatively low.Therefore, current data
The method of synchronization effectively using the synchronization process ability of synchronous processing equipment cannot have data synchronization efficiency
Relatively low problem.
The content of the invention
In view of the above problems, it is proposed that the embodiment of the present application overcomes above mentioned problem or extremely to provide one kind
A kind of method of data synchronization and a kind of corresponding data synchronization unit for partially solving the above problems.
In order to solve the above problems, this application discloses a kind of method of data synchronization, including：
According to the priority of the data volume of data to be synchronized and synchronous task in source database, fortune is expected in generation
Capable line synchro number of passes；
The line synchro number of passes for desirably running carries out thread configuration, and using the thread for configuring for described
Data syn-chronization to be synchronized is to target database.
Alternatively, the line synchro number of passes for desirably running carries out thread configuration includes：
Judge the line synchro number of passes for expecting operation whether more than the line synchro number of passes needed for operation；
If so, the line synchro number of passes needed for the line synchro number of passes for then being run according to the expectation and the operation
Difference, the corresponding thread of additional configurations.
Alternatively, described according to the preferential of the data volume of data to be synchronized in source database and synchronous task
Level, before the line synchro number of passes of operation is expected in generation, methods described also includes：
In the data syn-chronization task submitted to from user, the priority of the synchronous task is obtained, and from preservation
Need in the source database of synchrodata, obtain the data volume of the data to be synchronized.
Alternatively, it is described according to the preferential of the data volume of data to be synchronized in source database and synchronous task
Level, generation expects that the line synchro number of passes of operation includes：
Search the average synchronous speed for the data volume of the data to be synchronized and the priority of synchronous task
Rate；
Using the mean synchronization rate, the maximum no-load speed and the synchronization process of synchronous processing equipment
Equipment is available for the Thread Count of synchronization process, calculates the line synchro number of passes for expecting operation.
Alternatively, judge the line synchro number of passes for expecting operation whether more than same needed for running described
Before step Thread Count, methods described also includes：
From the target database for treating to carry out data syn-chronization with source database, needed for the acquisition operation
Line synchro number of passes.
Alternatively, in the thread using configuration for the data syn-chronization to be synchronized to target database
Before, methods described also includes：
The data to be synchronized are split into according to the line synchro number of passes needed for the operation multiple to be synchronized
Data block；
It is described use configuration thread for the data syn-chronization to be synchronized to target database for：
The synchronizing thread of synchronous each data block to be synchronized will be used for, synchronous processing equipment will be dispatched to, with by
The synchronous processing equipment processes the synchronizing thread.
Alternatively, the thread using configuration is for the data syn-chronization to be synchronized to target database bag
Include：
Thread attribute is met at least one pre-conditioned synchronizing thread preferentially to send to the synchronization
Reason equipment.
Alternatively, it is described pre-conditioned including following at least one：
At least one synchronizing thread belongs to same data syn-chronization task, pending time more than default
Time threshold, the priority of correspondence synchronous task are more than pre-set priority threshold value, single synchronous processing equipment
Can synchronization process.
Alternatively, the thread using configuration is for the data syn-chronization to be synchronized to target database bag
Include：
The synchronizing thread is preferentially sent to being available for the maximum thread of synchronization process to be more than default thread
The synchronous processing equipment of number threshold value.
In order to solve the above problems, disclosed herein as well is a kind of data synchronization unit, including：
Line synchro number of passes generation module, for according to the data volume of data to be synchronized and synchronization in source database
The line synchro number of passes of operation is expected in the priority of task, generation；
Synchronous processing module, the line synchro number of passes for desirably running carries out thread configuration, and uses
The thread of configuration is for the data syn-chronization to be synchronized to target database.
Alternatively, the synchronous processing module includes：
Line synchro number of passes judging submodule, for judging whether the line synchro number of passes for expecting operation is more than
Line synchro number of passes needed for operation, if so, then calling thread to supplement submodule；
Thread supplements submodule, for according to needed for the line synchro number of passes for expecting operation and the operation
Line synchro number of passes difference, the corresponding thread of additional configurations.
Alternatively, described device also includes：
Priority and data volume acquisition module, in the data syn-chronization task submitted to from user, obtaining institute
The priority of synchronous task is stated, and from the source database of data to be synchronized is preserved, treats same described in acquisition
The data volume of step data.
Alternatively, the line synchro number of passes generation module includes：
Mean synchronization rate searches submodule, for lookup for the data volume of the data to be synchronized and together
The mean synchronization rate of the priority of step task；
Line synchro number of passes calculating sub module, for using the mean synchronization rate, synchronous processing equipment
Maximum no-load speed and the synchronous processing equipment are available for the Thread Count of synchronization process, calculate the expectation fortune
Capable line synchro number of passes.
Alternatively, described device also includes：
Line synchro number of passes acquisition module needed for operation, for from the mesh for treating to be carried out with source database data syn-chronization
In mark database, the line synchro number of passes needed for obtaining the operation.
Alternatively, described device also includes：
Data to be synchronized split module, for treating same by described according to the line synchro number of passes needed for the operation
Step data splits into multiple data blocks to be synchronized；
The synchronous processing module specifically for：
The synchronizing thread of synchronous each data block to be synchronized will be used for, synchronous processing equipment will be dispatched to, with by
The synchronous processing equipment processes the synchronizing thread.
Alternatively, the synchronous processing module specifically for：
Thread attribute is met at least one pre-conditioned synchronizing thread preferentially to send to the synchronization
Reason equipment.
Alternatively, it is described pre-conditioned including following at least one：
At least one synchronizing thread belongs to same data syn-chronization task, pending time more than default
Time threshold, the priority of correspondence synchronous task are more than pre-set priority threshold value, single synchronous processing equipment
Can synchronization process.
Alternatively, the synchronous processing equipment include multiple, the synchronous processing module specifically for：
The synchronizing thread is preferentially sent to being available for the maximum thread of synchronization process to be more than default thread
The synchronous processing equipment of number threshold value.
The embodiment of the present application includes advantages below：
According to the embodiment of the present application, by according to the data volume of data to be synchronized and synchronization in source database
The line synchro number of passes of operation is expected in the priority generation of task, according to the line synchro number of passes for expecting operation
The thread for configuring respective numbers carries out data syn-chronization, such that it is able to be directed to the actual feelings of synchronous task
Condition dynamic configuration thread, it is to avoid part thread occur and completed synchronous task, and same task
Other threads improve the efficiency and stability of data syn-chronization also in the situation of wait state.And
And, dynamically the line synchro number of passes for running is expected in adjustment to the priority according to synchronous task so that important
Degree synchronous task higher can be processed preferentially.
Brief description of the drawings
The step of Fig. 1 is a kind of method of data synchronization embodiment one of the application flow chart；
The step of Fig. 2 is a kind of method of data synchronization embodiment two of the application flow chart；
Fig. 3 is a kind of structured flowchart of data synchronization unit embodiment one of the application；
Fig. 4 is a kind of structured flowchart of data synchronization unit embodiment two of the application.
Specific embodiment
To enable above-mentioned purpose, the feature and advantage of the application more obvious understandable, below in conjunction with the accompanying drawings
The application is described in further detail with specific embodiment.
Reference picture 1, flow chart the step of show a kind of method of data synchronization embodiment one of the application,
Specifically may include steps of：
Step 101, according to the priority of the data volume of data to be synchronized and synchronous task in source database,
The line synchro number of passes of operation is expected in generation.
It should be noted that the line synchro number of passes for expecting operation can be that the first run pre-set by user is same
Step Thread Count or many wheel line synchro numbers of passes.Because user cannot learn data to be synchronized in this synchronous task
Situation and synchronous processing equipment ruuning situation, or user and do not understand how set, it is set
Line synchro number of passes line synchro number of passes actually required when may be synchronous with operation is mismatched, same so as to cause
Synchronizing thread is distributed on busy synchronous processing equipment during step；Or user will expect the synchronization of operation
Thread Count is arranged to a fixed value, so as to cause synchronous task to cannot get rational synchronization process resource point
With and influence synchronous efficiency.Additionally, the important procedure of the synchronous task of different pieces of information is differed, and it is synchronous
Controller by each synchronous task fair play, cause important synchronizing thread because cannot get process resource and
Cannot priority synchronization.Therefore, data syn-chronization is carried out using current data syn-chronization mode, is may result in
Above-mentioned problem, so as to ultimately result in, data synchronization efficiency is relatively low, and the treatment of synchronous task also lacks stabilization
Property.
In the implementing of the embodiment of the present application, it is possible to use the data volume of data to be synchronized in source database
With the priority of synchronous task, the line synchro number of passes of generation expectation operation.In practical application, user can be with
Submit in a data synchronous task, the data syn-chronization task generally comprises the phase of synchronous task priority
Pass information, therefore can therefrom obtain the priority of synchronous task.Appoint for the data syn-chronization that user submits to
Business, from the source database of data source to be synchronized, can obtain the data volume of synchrodata.Certainly,
Those skilled in the art can obtain the information of data volume and priority etc. using other approach, such as in number
According to the information of priority of the data volume of insertion data to be synchronized and synchronous task etc. in synchronous task, so as to
In direct access data volume and priority from the data syn-chronization task submitted to.
Expect that the concrete mode of the line synchro number of passes of operation can have many according to data volume and priority generation
Kind.For example, a table of comparisons for data volume-priority-mean synchronization rate can be preset, in the table of comparisons
The corresponding specific mean synchronization rate of the combination of different data volume with priority.According to data volume and
Priority lookup is to corresponding mean synchronization rate.The mean synchronization rate that will be searched, sentences synchronization process
The synchronous speed that each CPU (Central Processing Unit, central processing unit) can run in equipment
Rate, it may be determined that the CPU quantity needed for synchronization, can be using the CPU quantity as the synchronization for expecting operation
Thread Count.
Step 102, the line synchro number of passes for desirably running carries out thread configuration, and using the line of configuration
Journey is for the data syn-chronization to be synchronized to target database.
The line synchro number of passes that can desirably run, configures to synchronizing thread.The mode of configuration can
It is various to have, for example can be actually required with synchronous operation synchronous according to the line synchro number of passes for expecting operation
The difference of Thread Count, configures corresponding virtual thread, using the virtual thread of configuration by synchronous processing equipment
On part CPU be placed in holding state and suspend process other synchronization task, to ensure that synchronization process sets
Have enough resources and process the data syn-chronization task；When the line synchro number of passes and synchronous operation of expecting operation
Actually required line synchro number of passes is identical, then the line synchro number of passes that directly can desirably run, configuration
Corresponding entity thread.
Can be using the entity thread of configuration and/or virtual thread synchronization process data to be synchronized.Practical application
In, data to be synchronized first can be split into multiple data blocks to be synchronized, the thread of each configuration is used respectively
In synchronous multiple data blocks to be synchronized.According to the remaining process resource of synchronous processing equipment, by multiple threads
One or more synchronous processing equipments are dispatched to, by synchronous processing equipment by multiple data block synchronizations to be synchronized
To target database.
When synchronizing thread is dispatched to synchronous processing equipment by isochronous controller, it can be determined that synchronous operation is real
Whether the line synchro number of passes needed for border is carried out not more than the line synchro number of passes for expecting operation according to judged result
With the scheduling of mode.
For example, when the actually required line synchro number of passes of synchronous operation is more than the line synchro number of passes for expecting operation,
Isochronous controller can be prepared for synchronous task configuration using the data syn-chronization pattern of multimachine multithreading
First round entity thread scheduling, it is preferential to send stand-by period most long, highest priority and belong to identical
The entity thread of synchronous task to multiple synchronous processing equipments, to ensure each thread of the synchronous task
Priority treatment can be obtained, it is to avoid certain thread has completed synchronization, and belong to synchronous task its together
His thread also in etc. pending state, have influence on the mean synchronization rate of the synchronous task.
The line synchro number of passes that expectation runs, synchronous control are less than when actually required line synchro number of passes is run simultaneously
Device processed can prepare the entity for synchronous task configuration using the data syn-chronization pattern of unit multithreading
The scheduling of thread and virtual thread, by the stand-by period is most long, highest priority and belongs to identical synchronous task
Thread, the preferential line synchro number of passes that can support to expect to run to some remaining process resource of sending
On synchronous processing equipment；If not meeting the synchronous processing equipment of condition, can with prioritizing selection residue at
A most synchronous processing equipment of reason resource is used as sending object.
Certainly, those skilled in the art can use unit single thread, multimachine single thread according to actual conditions
Deng data syn-chronization pattern be scheduled, can be carried out according to the process resource of synchronous processing equipment during scheduling
Load balance scheduling, it is also possible to which random schedule, the embodiment of the present application is not restricted to this.
According to the embodiment of the present application, by according to the data volume of data to be synchronized in source database and synchronous
The line synchro number of passes of operation is expected in the priority generation of business, and phase is configured according to the line synchro number of passes for expecting operation
Answering the thread of quantity carries out data syn-chronization, such that it is able to be directed to the actual conditions dynamic configuration of synchronous task
Thread, it is to avoid part thread occur and completed synchronous task, and other threads of same task also in
The situation of wait state, improves the efficiency and stability of data syn-chronization.And, according to synchronous task
Dynamically the line synchro number of passes of operation is expected in adjustment to priority so that significance level synchronous task higher can be with
Preferentially processed.
Reference picture 2, flow chart the step of show a kind of method of data synchronization embodiment two of the application,
Specifically may include steps of：
Step 201, in the data syn-chronization task submitted to from user, obtains the priority of the synchronous task,
And from the source database of data to be synchronized is preserved, obtain the data volume of the data to be synchronized.
In concrete implementation, when needing to carry out data syn-chronization, it is same that user can create data in client
Step task simultaneously submits to corresponding synchronization process server.When data syn-chronization task is created, client can
Corresponding priority is generated with according to the significance level of data to be synchronized, and is included in data syn-chronization task
In, therefore synchronization process server can obtain the preferential of synchronous task from the data syn-chronization task submitted to
Level.Further, it is also possible to read number to be synchronized from the metadata of the source database for preserving data to be synchronized
According to data volume.For example when being synchronized to Hbase from HDFS, treated by the acquisition of information for reading HDFS
The data volume of synchrodata, other kinds of database can also obtain data volume by similar mode.
Step 202, search for the data to be synchronized data volume and synchronous task priority it is flat
Equal no-load speed.
Step 203, using the mean synchronization rate, the maximum no-load speed of synchronous processing equipment and institute
The Thread Count that synchronous processing equipment is available for synchronization process is stated, the line synchro number of passes for expecting operation is calculated.
Can be searched synchronous according to the data volume of the data to be synchronized for getting and the priority of synchronous task
The mean synchronization rate of task.Specifically, can be previously according to data volume and synchronous of priority initialization
The expectation mean synchronization rate of business, for example：
Priority according to the data volume of data to be synchronized and synchronous task in current synchronous task is carried out
Search, for example, data volume is 0-10G, priority is 8 synchronous task, its correspondence can be found
Average synchronous task be 30mb/s.
Then according to the process resource according to each synchronous processing equipment, the CPU numbers needed for calculating synchronization
Amount.For example, each synchronous processing equipment has 6 CPU cores, its maximum no-load speed is 60mb/s,
The maximum no-load speed that each CPU can be then calculated can be run is 10mb/s.
The expectation mean synchronization rate 30mb/s of the synchronous task of lookup can be run divided by each CPU
Maximum no-load speed be 10mb/s, the CPU numbers needed for can calculating as expect fortune for 3
Capable line synchro number of passes.
Step 204, from treating to carry out the target database of data syn-chronization with source database, obtains the fortune
Line synchro number of passes needed for row.
Line synchro number of passes needed for operation can be read from the source data of target database, such as from HDFS
When being synchronized to HBase, the Region numbers of HBase object tables can be read, with Hbase object tables
Region numbers as the line synchro number of passes needed for operation.Region is HBase data storages and pipe
The base unit of reason, generally determines the Thread Count that target database is used in synchronization by Region.
Whether step 205, judge the line synchro number of passes for expecting operation more than the line synchro needed for operation
Number of passes；If so, then carrying out step 206.
Step 206, the synchronizing thread according to needed for the line synchro number of passes for expecting operation and the operation
Several differences, the corresponding thread of additional configurations.
When the line synchro number of passes for the expecting operation line synchro number of passes actually required more than synchronous operation, can be with root
The difference of the line synchro number of passes needed for subtracting operation according to the line synchro number of passes for expecting operation, additional configurations are corresponding
Virtual thread, using configuration virtual thread the part CPU on synchronous processing equipment is placed in it is standby
State simultaneously suspends process other synchronization task, to ensure that synchronous processing equipment has enough resources to process the number
According to synchronous task.
Step 207, splits into the data to be synchronized according to the line synchro number of passes needed for the operation many
Individual data block to be synchronized.
Step 208, using the thread of configuration for the data syn-chronization to be synchronized to target database.
Used as the preferred exemplary of the embodiment of the present application, the step 208 can be specially：
The synchronizing thread of synchronous each data block to be synchronized will be used for, synchronous processing equipment will be dispatched to, with by
The synchronous processing equipment processes the synchronizing thread.
Data to be synchronized can be split into multiple data blocks to be synchronized, the thread of each configuration is respectively used to
Synchronous multiple data blocks to be synchronized.According to the remaining process resource of synchronous processing equipment, multiple threads are adjusted
Be synchronized to for multiple data blocks to be synchronized by synchronous processing equipment to one or more synchronous processing equipments by degree
Target database.
In actual application, the corresponding multiple threads of data syn-chronization task can be committed to Synchronization Control
Whether device, carried out expecting the line synchro number of passes of operation more than the synchronizing thread needed for operation as isochronous controller
Several judgement treatment, and according to judged result correspondingly additional configurations thread and by thread scheduling to multiple same
Step processing equipment.
Used as the preferred exemplary of the embodiment of the present application, the step 208 can include：
Thread attribute is met at least one pre-conditioned synchronizing thread preferentially to send to the synchronization
Reason equipment；Wherein, it is described pre-conditioned to include following at least one：
At least one synchronizing thread belongs to same data syn-chronization task, pending time more than default
Time threshold, the priority of correspondence synchronous task are more than pre-set priority threshold value, single synchronous processing equipment
Can synchronization process.
For the feelings for expecting the line synchro number of passes of the operation line synchro number of passes actually required more than synchronous operation
Condition, isochronous controller can prepare to be directed to the synchronous task using the data syn-chronization pattern of unit multithreading
The entity thread of configuration and the scheduling of virtual thread, by the pending time more than preset time threshold, preferential
Level is more than pre-set priority threshold value and belongs to the thread of same data syn-chronization task, preferential to send to single
Synchronous processing equipment can be on the synchronous processing equipment of synchronization process；If not meeting the synchronization process of condition
Equipment, can be using the prioritizing selection most synchronous processing equipment of residue process resource as sending object.
Used as the preferred exemplary of the embodiment of the present application, the synchronous processing equipment includes multiple, the step
208 can include：
The synchronizing thread is preferentially sent to being available for the maximum thread of synchronization process to be more than default thread
The synchronous processing equipment of number threshold value.
In practical application, can temporarily may not meet single synchronous processing equipment can synchronization process this is pre-
If the synchronous processing equipment of condition, now synchronizing thread can be preferentially sent to being available for synchronization process most
Big Thread Count is more than on multiple synchronous processing equipments of default Thread Count threshold value, that is, select at CPU residues
The more synchronous processing equipment of reason resource carries out the data syn-chronization of the first run or many wheels.For multiple synchronization process
The scheduling of equipment, can carry out load balance scheduling according to the process resource of synchronous processing equipment, it is also possible to
Random schedule, the embodiment of the present application is not restricted to this.
According to the embodiment of the present application, the synchronizing thread according to needed for the line synchro number of passes and operation of expecting operation
Several corresponding virtual thread of difference additional configurations, using virtual thread by the part on synchronous processing equipment
CPU is placed in holding state and suspends process other synchronization task, to ensure that it is enough that synchronous processing equipment has
Resource processes the data syn-chronization task, improves the efficiency and stability of data syn-chronization.Also, it is not necessary to
The use of CPU is controlled from system level, the flexibility of data syn-chronization is improved.
It should be noted that for embodiment of the method, in order to be briefly described, therefore it is all expressed as one it is
The combination of actions of row, but those skilled in the art should know, and the embodiment of the present application is not by described
Sequence of movement limitation because according to the embodiment of the present application, some steps can using other orders or
Person is carried out simultaneously.Secondly, those skilled in the art should also know, embodiment described in this description
Preferred embodiment is belonged to, necessary to involved action not necessarily the embodiment of the present application.
Reference picture 3, shows a kind of structured flowchart of data synchronization unit embodiment one of the application, tool
Body can include such as lower module：
Line synchro number of passes generation module 301, for according to the data volume of data to be synchronized in source database and
The line synchro number of passes of operation is expected in the priority of synchronous task, generation.
Synchronous processing module 302, the line synchro number of passes for desirably running carries out thread configuration, and
Using the thread of configuration for the data syn-chronization to be synchronized to target database.
According to the embodiment of the present application, by according to the data volume of data to be synchronized in source database and synchronous
The line synchro number of passes of operation is expected in the priority generation of business, and phase is configured according to the line synchro number of passes for expecting operation
Answering the thread of quantity carries out data syn-chronization, such that it is able to be directed to the actual conditions dynamic configuration of synchronous task
Thread, it is to avoid part thread occur and completed synchronous task, and other threads of same task also in
The situation of wait state, improves the efficiency and stability of data syn-chronization.And, according to synchronous task
Dynamically the line synchro number of passes of operation is expected in adjustment to priority so that significance level synchronous task higher can be with
Preferentially processed.
Reference picture 4, shows a kind of structured flowchart of data synchronization unit embodiment two of the application, tool
Body can include such as lower module：
Priority and data volume acquisition module 401, in the data syn-chronization task submitted to from user, obtaining
The priority of the synchronous task is taken, and from the source database of data to be synchronized is preserved, obtains described
The data volume of data to be synchronized.
Line synchro number of passes acquisition module 402 needed for operation, for from treating to carry out data syn-chronization with source database
Target database in, the line synchro number of passes needed for obtaining the operation.
Line synchro number of passes generation module 403, for according to the data volume of data to be synchronized in source database and
The line synchro number of passes of operation is expected in the priority of synchronous task, generation.
Synchronous processing module 404, the line synchro number of passes for desirably running carries out thread configuration, and
Using the thread of configuration for the data syn-chronization to be synchronized to target database.
Used as the preferred exemplary of the embodiment of the present application, the line synchro number of passes generation module 403 can include：
Mean synchronization rate searches submodule, for lookup for the data volume of the data to be synchronized and together
The mean synchronization rate of the priority of step task；
Line synchro number of passes calculating sub module, for using the mean synchronization rate, synchronous processing equipment
Maximum no-load speed and the synchronous processing equipment are available for the Thread Count of synchronization process, calculate the expectation fortune
Capable line synchro number of passes.
Used as the preferred exemplary of the embodiment of the present application, the synchronous processing module 404 can include：
Line synchro number of passes judging submodule, for judging whether the line synchro number of passes for expecting operation is more than
Line synchro number of passes needed for operation, if so, then calling thread to supplement submodule.
Thread supplements submodule, for according to needed for the line synchro number of passes for expecting operation and the operation
Line synchro number of passes difference, the corresponding thread of additional configurations.
Used as the preferred exemplary of the embodiment of the present application, described device can also include：
Data to be synchronized split module, for treating same by described according to the line synchro number of passes needed for the operation
Step data splits into multiple data blocks to be synchronized.
Used as the preferred exemplary one of the embodiment of the present application, the synchronous processing module 404 can be specifically for：
The synchronizing thread of synchronous each data block to be synchronized will be used for, synchronous processing equipment will be dispatched to, with by
The synchronous processing equipment processes the synchronizing thread.
Used as the preferred exemplary two of the embodiment of the present application, the synchronous processing module 404 can be specifically for：
Thread attribute is met at least one pre-conditioned synchronizing thread preferentially to send to the synchronization
Reason equipment.
It is described pre-conditioned including following at least one as the preferred exemplary of the embodiment of the present application：
At least one synchronizing thread belongs to same data syn-chronization task, pending time more than default
Time threshold, the priority of correspondence synchronous task are more than pre-set priority threshold value, single synchronous processing equipment
Can synchronization process.
Used as the preferred exemplary three of the embodiment of the present application, the synchronous processing equipment includes multiple, described same
Step processing module 404 can be specifically for：
The synchronizing thread is preferentially sent to being available for the maximum thread of synchronization process to be more than default thread
The synchronous processing equipment of number threshold value.
According to the embodiment of the present application, the synchronizing thread according to needed for the line synchro number of passes and operation of expecting operation
Several corresponding virtual thread of difference additional configurations, using virtual thread by the part on synchronous processing equipment
CPU is placed in holding state and suspends process other synchronization task, to ensure that it is enough that synchronous processing equipment has
Resource processes the data syn-chronization task, improves the efficiency and stability of data syn-chronization.Also, it is not necessary to
The use of CPU is controlled from system level, the flexibility of data syn-chronization is improved.
For device embodiment, because it is substantially similar to embodiment of the method, so the comparing of description
Simply, the relevent part can refer to the partial explaination of embodiments of method.
Each embodiment in this specification is described by the way of progressive, and each embodiment is stressed
Be all difference with other embodiment, between each embodiment identical similar part mutually referring to
.
It should be understood by those skilled in the art that, the embodiment of the embodiment of the present application can be provided as method, dress
Put or computer program product.Therefore, the embodiment of the present application can using complete hardware embodiment, completely
The form of the embodiment in terms of software implementation or combination software and hardware.And, the embodiment of the present application
Can use can be situated between in one or more computers for wherein including computer usable program code with storage
The computer journey implemented in matter (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of sequence product.
In a typical configuration, the computer equipment includes one or more processors
(CPU), input/output interface, network interface and internal memory.Internal memory potentially includes computer-readable medium
In volatile memory, the shape such as random access memory (RAM) and/or Nonvolatile memory
Formula, such as read-only storage (ROM) or flash memory (flash RAM).Internal memory is computer-readable medium
Example.Computer-readable medium includes permanent and non-permanent, removable and non-removable media
Information Store can be realized by any method or technique.Information can be computer-readable instruction,
Data structure, the module of program or other data.The example of the storage medium of computer includes, but
Phase transition internal memory (PRAM), static RAM (SRAM), dynamic random is not limited to deposit
Access to memory (DRAM), other kinds of random access memory (RAM), read-only storage
(ROM), Electrically Erasable Read Only Memory (EEPROM), fast flash memory bank or other in
Deposit technology, read-only optical disc read-only storage (CD-ROM), digital versatile disc (DVD) or other
Optical storage, magnetic cassette tape, tape magnetic rigid disk storage other magnetic storage apparatus or it is any its
His non-transmission medium, can be used to store the information that can be accessed by a computing device.According to herein
Define, computer-readable medium does not include the computer readable media (transitory media) of non-standing,
Such as the data-signal and carrier wave of modulation.
The embodiment of the present application is with reference to the method according to the embodiment of the present application, terminal device (system) and meter
The flow chart and/or block diagram of calculation machine program product is described.It should be understood that can be by computer program instructions
Realize each flow and/or square frame and flow chart and/or the square frame in flow chart and/or block diagram
The combination of flow and/or square frame in figure.Can provide these computer program instructions to all-purpose computer,
The processor of special-purpose computer, Embedded Processor or other programmable data processing terminal equipments is producing
One machine so that by the computing device of computer or other programmable data processing terminal equipments
Instruction produce for realizing in one flow of flow chart or multiple one square frame of flow and/or block diagram or
The device of the function of being specified in multiple square frames.
These computer program instructions may be alternatively stored in can guide computer or other programmable datas to process
In the computer-readable memory that terminal device works in a specific way so that storage is in the computer-readable
Instruction in memory is produced and includes the manufacture of command device, and command device realization is in flow chart one
The function of being specified in flow or multiple one square frame of flow and/or block diagram or multiple square frames.
These computer program instructions can also be loaded into computer or other programmable data processing terminals set
It is standby upper so that execution series of operation steps is in terms of producing on computer or other programmable terminal equipments
The treatment that calculation machine is realized, so as to the instruction performed on computer or other programmable terminal equipments provides use
In realization in one flow of flow chart or multiple one square frame of flow and/or block diagram or multiple square frames
The step of function of specifying.
Although having been described for the preferred embodiment of the embodiment of the present application, those skilled in the art are once
Basic creative concept is known, then other change and modification can be made to these embodiments.So,
Appended claims are intended to be construed to include preferred embodiment and fall into the institute of the embodiment of the present application scope
Have altered and change.
Finally, in addition it is also necessary to explanation, herein, such as first and second or the like relational terms
It is used merely to make a distinction an entity or operation with another entity or operation, and not necessarily requires
Or imply between these entities or operation there is any this actual relation or order.And, art
Language " including ", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion so that
Process, method, article or terminal device including a series of key elements not only include those key elements, and
Also include other key elements for being not expressly set out, or also include for this process, method, article or
The intrinsic key element of person's terminal device.In the absence of more restrictions, by sentence " including
It is individual ... " limit key element, it is not excluded that at the process including the key element, method, article or end
Also there is other identical element in end equipment.
Above to a kind of method of data synchronization and a kind of data synchronization unit provided herein, carry out
It is discussed in detail, specific case used herein is set forth to the principle and implementation method of the application,
The explanation of above example is only intended to help and understands the present processes and its core concept；Meanwhile, it is right
In those of ordinary skill in the art, according to the thought of the application, in specific embodiment and range of application
On will change, in sum, this specification content should not be construed as the limitation to the application.
Claims (18)
1. a kind of method of data synchronization, it is characterised in that including：
According to the priority of the data volume of data to be synchronized and synchronous task in source database, fortune is expected in generation
Capable line synchro number of passes；
The line synchro number of passes for desirably running carries out thread configuration, and using the thread for configuring for described
Data syn-chronization to be synchronized is to target database.
2. method according to claim 1, it is characterised in that it is described desirably run it is same
Step Thread Count carries out thread configuration to be included：
Judge the line synchro number of passes for expecting operation whether more than the line synchro number of passes needed for operation；
If so, the line synchro number of passes needed for the line synchro number of passes for then being run according to the expectation and the operation
Difference, the corresponding thread of additional configurations.
3. method according to claim 1, it is characterised in that described according in source database
The data volume of data to be synchronized and the priority of synchronous task, generation expect operation line synchro number of passes it
Before, methods described also includes：
In the data syn-chronization task submitted to from user, the priority of the synchronous task is obtained, and from preservation
Need in the source database of synchrodata, obtain the data volume of the data to be synchronized.
4. method according to claim 1, it is characterised in that described to be treated according in source database
The data volume of synchrodata and the priority of synchronous task, generation expect that the line synchro number of passes of operation includes：
Search the average synchronous speed for the data volume of the data to be synchronized and the priority of synchronous task
Rate；
Using the mean synchronization rate, the maximum no-load speed and the synchronization process of synchronous processing equipment
Equipment is available for the Thread Count of synchronization process, calculates the line synchro number of passes for expecting operation.
5. method according to claim 2, it is characterised in that judge described to expect fortune described
Before whether capable line synchro number of passes is more than the line synchro number of passes needed for operation, methods described also includes：
From the target database for treating to carry out data syn-chronization with source database, needed for the acquisition operation
Line synchro number of passes.
6. method according to claim 1, it is characterised in that in the thread using configuration
Before for the data syn-chronization to be synchronized to target database, methods described also includes：
The data to be synchronized are split into according to the line synchro number of passes needed for the operation multiple to be synchronized
Data block；
It is described use configuration thread for the data syn-chronization to be synchronized to target database for：
The synchronizing thread of synchronous each data block to be synchronized will be used for, synchronous processing equipment will be dispatched to, with by
The synchronous processing equipment processes the synchronizing thread.
7. method according to claim 1, it is characterised in that the thread pin using configuration
The data syn-chronization to be synchronized to target database is included：
Thread attribute is met at least one pre-conditioned synchronizing thread preferentially to send to the synchronization
Reason equipment.
8. method according to claim 7, it is characterised in that described pre-conditioned including following
It is at least one：
At least one synchronizing thread belongs to same data syn-chronization task, pending time more than default
Time threshold, the priority of correspondence synchronous task are more than pre-set priority threshold value, single synchronous processing equipment
Can synchronization process.
9. method according to claim 1, it is characterised in that the synchronous processing equipment includes
Multiple, the thread using configuration includes for the data syn-chronization to be synchronized to target database：
The synchronizing thread is preferentially sent to being available for the maximum thread of synchronization process to be more than default thread
The synchronous processing equipment of number threshold value.
10. a kind of data synchronization unit, it is characterised in that including：
Line synchro number of passes generation module, for according to the data volume of data to be synchronized and synchronization in source database
The line synchro number of passes of operation is expected in the priority of task, generation；
Synchronous processing module, the line synchro number of passes for desirably running carries out thread configuration, and uses
The thread of configuration is for the data syn-chronization to be synchronized to target database.
11. devices according to claim 10, it is characterised in that the synchronous processing module bag
Include：
Line synchro number of passes judging submodule, for judging whether the line synchro number of passes for expecting operation is more than
Line synchro number of passes needed for operation, if so, then calling thread to supplement submodule；
Thread supplements submodule, for according to needed for the line synchro number of passes for expecting operation and the operation
Line synchro number of passes difference, the corresponding thread of additional configurations.
12. devices according to claim 10, it is characterised in that described device also includes：
Priority and data volume acquisition module, in the data syn-chronization task submitted to from user, obtaining institute
The priority of synchronous task is stated, and from the source database of data to be synchronized is preserved, treats same described in acquisition
The data volume of step data.
13. devices according to claim 10, it is characterised in that the line synchro number of passes generation
Module includes：
Mean synchronization rate searches submodule, for lookup for the data volume of the data to be synchronized and together
The mean synchronization rate of the priority of step task；
Line synchro number of passes calculating sub module, for using the mean synchronization rate, synchronous processing equipment
Maximum no-load speed and the synchronous processing equipment are available for the Thread Count of synchronization process, calculate the expectation fortune
Capable line synchro number of passes.
14. devices according to claim 11, it is characterised in that described device also includes：
Line synchro number of passes acquisition module needed for operation, for from the mesh for treating to be carried out with source database data syn-chronization
In mark database, the line synchro number of passes needed for obtaining the operation.
15. devices according to claim 10, it is characterised in that described device also includes：
Data to be synchronized split module, for treating same by described according to the line synchro number of passes needed for the operation
Step data splits into multiple data blocks to be synchronized；
The synchronous processing module specifically for：
The synchronizing thread of synchronous each data block to be synchronized will be used for, synchronous processing equipment will be dispatched to, with by
The synchronous processing equipment processes the synchronizing thread.
16. devices according to claim 10, it is characterised in that the synchronous processing module tool
Body is used for：
Thread attribute is met at least one pre-conditioned synchronizing thread preferentially to send to the synchronization
Reason equipment.
17. devices according to claim 16, it is characterised in that it is described it is pre-conditioned including with
Lower at least one：
At least one synchronizing thread belongs to same data syn-chronization task, pending time more than default
Time threshold, the priority of correspondence synchronous task are more than pre-set priority threshold value, single synchronous processing equipment
Can synchronization process.
18. devices according to claim 10, it is characterised in that the synchronous processing equipment bag
Include multiple, the synchronous processing module specifically for：
The synchronizing thread is preferentially sent to being available for the maximum thread of synchronization process to be more than default thread
The synchronous processing equipment of number threshold value.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511032646.9A CN106933534B (en) | 2015-12-31 | 2015-12-31 | Data synchronization method and device |
| PCT/CN2016/110658 WO2017114199A1 (en) | 2015-12-31 | 2016-12-19 | Data synchronisation method and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511032646.9A CN106933534B (en) | 2015-12-31 | 2015-12-31 | Data synchronization method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106933534A true CN106933534A (en) | 2017-07-07 |
| CN106933534B CN106933534B (en) | 2020-07-28 |
Family
ID=59224514
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511032646.9A Active CN106933534B (en) | 2015-12-31 | 2015-12-31 | Data synchronization method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106933534B (en) |
| WO (1) | WO2017114199A1 (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107402804A (en) * | 2017-07-31 | 2017-11-28 | 广东欧珀移动通信有限公司 | Background process management-control method, device, storage medium and electronic equipment |
| CN111382199A (en) * | 2018-12-29 | 2020-07-07 | 中兴通讯股份有限公司 | Method and device for synchronously copying database |
| CN111858744A (en) * | 2019-04-29 | 2020-10-30 | 北京嘀嘀无限科技发展有限公司 | Database data synchronization method, server and system |
| CN112148793A (en) * | 2020-09-17 | 2020-12-29 | 睿住科技有限公司 | Data synchronization method, system and storage medium |
| CN112182100A (en) * | 2020-09-22 | 2021-01-05 | 烽火通信科技股份有限公司 | Cloud management platform state data synchronization method and system |
| CN112818054A (en) * | 2020-10-15 | 2021-05-18 | 广州南天电脑系统有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN114237505A (en) * | 2021-12-14 | 2022-03-25 | 中国建设银行股份有限公司 | Batch processing method and device of business data and computer equipment |
| CN116225660A (en) * | 2023-03-17 | 2023-06-06 | 中国华能集团有限公司北京招标分公司 | Loading engine using method |
| CN116708480A (en) * | 2023-07-27 | 2023-09-05 | 深圳迅策科技有限公司 | Data synchronization method based on Datax frame |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377402A (en) * | 2018-04-12 | 2019-10-25 | 腾讯科技（深圳）有限公司 | Business data processing method, device, storage medium and computer equipment |
| CN109241175B (en) * | 2018-06-28 | 2021-06-04 | 东软集团股份有限公司 | Data synchronization method and device, storage medium and electronic equipment |
| CN110225077A (en) * | 2019-04-25 | 2019-09-10 | 深圳壹账通智能科技有限公司 | Synchronous method, device, computer equipment and the computer storage medium of change supply data |
| CN110334018A (en) * | 2019-06-18 | 2019-10-15 | 梁俊杰 | A kind of big data introduction method and relevant device |
| CN111046091B (en) * | 2019-10-24 | 2023-12-08 | 杭州数梦工场科技有限公司 | Operation method, device and equipment of data exchange system |
| CN111190961B (en) * | 2019-12-18 | 2023-09-29 | 航天信息股份有限公司 | Dynamic optimization multithreading data synchronization method and system |
| CN112162964B (en) * | 2020-10-15 | 2024-04-05 | 苏州交驰人工智能研究院有限公司 | Self-adaptive data synchronization method, device, computer equipment and storage medium |
| CN116048780B (en) * | 2022-12-07 | 2023-08-08 | 广州海量数据库技术有限公司 | Multithreading batch file loading method based on openGauss database |
| CN116303702B (en) * | 2022-12-27 | 2024-04-05 | 易方达基金管理有限公司 | ETL-based data parallel processing method, device, equipment and storage medium |
| CN116204587B (en) * | 2023-02-21 | 2024-01-30 | 中国人民解放军海军工程大学 | Data synchronization task generation method, device and computer readable storage medium |
| CN116257365B (en) * | 2023-05-15 | 2023-08-22 | 建信金融科技有限责任公司 | Data warehousing method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070008890A1 (en) * | 2005-07-11 | 2007-01-11 | Tseitlin Eugene R | Method and apparatus for non-stop multi-node system synchronization |
| CN103136035A (en) * | 2011-11-30 | 2013-06-05 | 国际商业机器公司 | Method and device for thread management of hybrid threading mode program |
| CN103678718A (en) * | 2013-12-31 | 2014-03-26 | 金蝶软件（中国）有限公司 | Database synchronization method and system |
| CN103778136A (en) * | 2012-10-19 | 2014-05-07 | 阿里巴巴集团控股有限公司 | Cross-room database synchronization method and system |
| CN103942259A (en) * | 2014-03-21 | 2014-07-23 | 浙江大学 | Method for achieving data caching in database synchronization process |
-
2015
- 2015-12-31 CN CN201511032646.9A patent/CN106933534B/en active Active
-
2016
- 2016-12-19 WO PCT/CN2016/110658 patent/WO2017114199A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070008890A1 (en) * | 2005-07-11 | 2007-01-11 | Tseitlin Eugene R | Method and apparatus for non-stop multi-node system synchronization |
| CN103136035A (en) * | 2011-11-30 | 2013-06-05 | 国际商业机器公司 | Method and device for thread management of hybrid threading mode program |
| CN103778136A (en) * | 2012-10-19 | 2014-05-07 | 阿里巴巴集团控股有限公司 | Cross-room database synchronization method and system |
| CN103678718A (en) * | 2013-12-31 | 2014-03-26 | 金蝶软件（中国）有限公司 | Database synchronization method and system |
| CN103942259A (en) * | 2014-03-21 | 2014-07-23 | 浙江大学 | Method for achieving data caching in database synchronization process |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107402804A (en) * | 2017-07-31 | 2017-11-28 | 广东欧珀移动通信有限公司 | Background process management-control method, device, storage medium and electronic equipment |
| CN107402804B (en) * | 2017-07-31 | 2019-12-10 | Oppo广东移动通信有限公司 | background process control method and device, storage medium and electronic equipment |
| CN111382199A (en) * | 2018-12-29 | 2020-07-07 | 中兴通讯股份有限公司 | Method and device for synchronously copying database |
| CN111858744A (en) * | 2019-04-29 | 2020-10-30 | 北京嘀嘀无限科技发展有限公司 | Database data synchronization method, server and system |
| CN112148793A (en) * | 2020-09-17 | 2020-12-29 | 睿住科技有限公司 | Data synchronization method, system and storage medium |
| CN112148793B (en) * | 2020-09-17 | 2024-02-20 | 广东睿住智能科技有限公司 | Data synchronization method, system and storage medium |
| CN112182100B (en) * | 2020-09-22 | 2022-11-08 | 烽火通信科技股份有限公司 | Cloud management platform state data synchronization method and system |
| CN112182100A (en) * | 2020-09-22 | 2021-01-05 | 烽火通信科技股份有限公司 | Cloud management platform state data synchronization method and system |
| CN112818054B (en) * | 2020-10-15 | 2022-05-06 | 广州南天电脑系统有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN112818054A (en) * | 2020-10-15 | 2021-05-18 | 广州南天电脑系统有限公司 | Data synchronization method and device, computer equipment and storage medium |
| CN114237505A (en) * | 2021-12-14 | 2022-03-25 | 中国建设银行股份有限公司 | Batch processing method and device of business data and computer equipment |
| CN116225660A (en) * | 2023-03-17 | 2023-06-06 | 中国华能集团有限公司北京招标分公司 | Loading engine using method |
| CN116225660B (en) * | 2023-03-17 | 2024-02-27 | 中国华能集团有限公司北京招标分公司 | Loading engine using method |
| CN116708480A (en) * | 2023-07-27 | 2023-09-05 | 深圳迅策科技有限公司 | Data synchronization method based on Datax frame |
| CN116708480B (en) * | 2023-07-27 | 2023-09-29 | 深圳迅策科技有限公司 | Data synchronization method based on Datax frame |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106933534B (en) | 2020-07-28 |
| WO2017114199A1 (en) | 2017-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106933534A (en) | A kind of method of data synchronization and device | |
| US9846589B2 (en) | Virtual machine placement optimization with generalized organizational scenarios | |
| US9086902B2 (en) | Sending tasks between virtual machines based on expiration times | |
| Pandey et al. | Prominence of mapreduce in big data processing | |
| CN108959510B (en) | Partition level connection method and device for distributed database | |
| US9195701B2 (en) | System and method for flexible distributed massively parallel processing (MPP) database | |
| Turk et al. | Temporal workload-aware replicated partitioning for social networks | |
| CN106611037A (en) | Method and device for distributed diagram calculation | |
| CN104765589A (en) | Grid parallel preprocessing method based on MPI | |
| CN112286917B (en) | Data processing method and device, electronic equipment and storage medium | |
| US11630986B2 (en) | Graph conversion method | |
| Park et al. | Processing of probabilistic skyline queries using mapreduce | |
| Gandomi et al. | HybSMRP: a hybrid scheduling algorithm in Hadoop MapReduce framework | |
| Bala et al. | P-ETL: Parallel-ETL based on the MapReduce paradigm | |
| Wong et al. | Parallel analytics as a service | |
| Yassien et al. | RDBMS, NoSQL, Hadoop: a performance-based empirical analysis | |
| Sax et al. | Performance optimization for distributed intra-node-parallel streaming systems | |
| Potter et al. | Dynamic data exchange in distributed RDF stores | |
| CN103338246B (en) | Virtual machine system of selection in a kind of infrastructure cloud resource allocation process and system | |
| CN105989151B (en) | Webpage capture method and device | |
| CN110196879B (en) | Data processing method, device, computing equipment and storage medium | |
| Lynden et al. | Dynamic data redistribution for MapReduce joins | |
| Fang et al. | Cost-effective stream join algorithm on cloud system | |
| CN110046028A (en) | Data processing method, device and server | |
| CN110046173B (en) | Method and device for generating scheduling information and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211108 Address after: Room 701-26, 7th floor, 2 Haidian East 3rd Street, Haidian District, Beijing Patentee after: YOUMENG TONGXIN (BEIJING) TECHNOLOGY CO.,LTD. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Patentee before: ALIBABA GROUP HOLDING Ltd. |
|
| TR01 | Transfer of patent right |