CN106209974B - A kind of method of data synchronization, equipment and system - Google Patents
A kind of method of data synchronization, equipment and system Download PDFInfo
- Publication number
- CN106209974B CN106209974B CN201610451188.0A CN201610451188A CN106209974B CN 106209974 B CN106209974 B CN 106209974B CN 201610451188 A CN201610451188 A CN 201610451188A CN 106209974 B CN106209974 B CN 106209974B
- Authority
- CN
- China
- Prior art keywords
- data
- data block
- block
- directory
- check code
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000005540 biological transmission Effects 0.000 claims abstract description 52
- 238000005070 sampling Methods 0.000 claims abstract description 27
- 238000005192 partition Methods 0.000 claims description 24
- 230000008859 change Effects 0.000 claims description 7
- 238000012512 characterization method Methods 0.000 claims description 7
- 239000002245 particle Substances 0.000 claims description 7
- 230000001360 synchronised effect Effects 0.000 claims description 4
- 241000239290 Araneae Species 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 5
- 238000012795 verification Methods 0.000 description 5
- 241001269238 Data Species 0.000 description 4
- 101150096185 PAAS gene Proteins 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
- H04L67/5651—Reducing the amount or size of exchanged application data
Abstract
The present invention provides a kind of method of data synchronization, equipment and system, method includes: that the corresponding sampling period is respectively configured for different types of service；When reaching the target type service corresponding sampling period, the data transmitted needed for the target type service are acquired；The data of the required transmission are divided at least one data block；It determines the first data block not being transmitted across at least one described data block, the second data block having been transmitted through, and determines the second data directory of the first data directory of first data block, second data block；By first data block, first data directory and second data directory, it is sent to data center, so that the data center stores at least one described data block.According to this programme, it is possible to reduce the data volume in transmission process, and then the occupancy of network bandwidth can be reduced.
Description
Technical field
The present invention relates to field of cloud computer technology, in particular to a kind of method of data synchronization, equipment and system.
Background technique
Cloud computing technology is utilized virtualization technology and the resources such as calculating, storage, network is carried out pond, will by internet
Shared software and hardware is supplied to user in a manner of servicing.Wherein, (Platform as a Service, platform take PAAS
Business) a kind of service type of the platform as cloud computing, it is supplied to software development on demand using software deployment, O&M as a kind of service
User becomes research direction very popular in recent years.
In PAAS platform, in order to realize system service and all kinds of customizing services, need to produce in each physical machine
Raw data synchronize storage.When the data generated on to each physical machine carry out real-time synchronization, physical machine passes through acquisition
The data are sent to data center by Internet resources, are stored that data by data center by the data generated to itself
In real-time data base.
Internet resources are abnormal precious for PAAS platform, and can generate largely in each physical machine in the prior art
Data cause the occupancy of network bandwidth larger.
Summary of the invention
The embodiment of the invention provides a kind of method of data synchronization, equipment and systems, to reduce the occupancy of network bandwidth.
In a first aspect, being applied to physical machine, for different type the embodiment of the invention provides a kind of method of data synchronization
Service the corresponding sampling period is respectively configured；The described method includes:
When reaching the target type service corresponding sampling period, the number transmitted needed for the target type service is acquired
According to；
The data of the required transmission are divided at least one data block；
Determine the first data block not being transmitted across at least one described data block, the second data having been transmitted through
Block, and determine the second data directory of the first data directory of first data block, second data block；
By first data block, first data directory and second data directory, it is sent to data center, with
The data center is set to store at least one described data block.
Preferably,
Before the data of the required transmission are divided at least one data block, further comprise: public using first
Formula computation partition length；
First formula includes:
Wherein, k (Si) for characterizing SiCorresponding partition length；SiI-th of the service provided for characterization platform；
period(si) for characterizing to SiThe sampling period of configuration；status(si) for characterizing siState change particle size；λ
It is known constant for indicating impact factor；Defatult_size is used for the default-length of characterize data block；
The data of the required transmission are divided at least one data block, comprising: the partition length is utilized, it will be required
The data of transmission are divided at least one described data block；
And/or
By first data block, first data directory and second data directory, it is sent to data center
Later, further comprise: each data block at least one described data block and at least one described data block is corresponding
Data directory be locally stored；
And/or
The first data block for not being transmitted across at least one data block described in the determination, second had been transmitted through
Data block, comprising:
Determine corresponding first check code of each data block in each data block being locally stored；
Calculate corresponding second check code of each data block at least one described data block；
According to second check code, first check code is traversed；Described the in first check code will be located at
The corresponding data block of two check codes is as second data block, second verification that will be not located in first check code
The corresponding data block of code is as first data block.
Preferably, corresponding second check code of each data block at least one data block described in the calculating, comprising:
Each data block pair at least one described data block is calculated using the second formula, third formula and the 4th formula
The second check code answered；
Second formula includes:
The third formula includes:
4th formula includes:
Adler32 (1, k)=A (1, k)+216B(1,k)；
Wherein, Adler32 (1, k) is for characterizing the second check code；A (1, k) is for characterizing the first intermediate parameters, B (1, k)
For characterizing the second intermediate parameters；Data [j] is used to characterize the corresponding data of j-th of byte in current data block；M is used for table
Levy known constant；K is for characterizing the byte number that the current data block includes.
Second aspect, the embodiment of the invention also provides a kind of method of data synchronization, are applied to data center, comprising:
It is corresponding to obtain the first data block, first data block for target type service that present physical machine is sent
Corresponding second data directory of first data directory, the second data block；Wherein, first data block is the present physical machine
The data block not being transmitted across, second data block are the data block that the present physical machine has been transmitted through；
According to each third data block for including in the data copy being locally stored and each described third data block
Corresponding third data directory, and according to second data directory, determine second data block；
Store first data block and second data block.
Preferably, further comprise: to first data block and second data block of storage, carrying out following target
The data copy of quantity replicates:
Wherein, number, which is used to characterize, carries out data copy duplication to first data block and second data block
The destination number；Request_dead_lock is for characterizing multi-user's request to first data block and second number
The quantity of service block is caused according to the data contention of block；All_request is corresponding simultaneously for characterizing the target type service
Send out amount of access；A is known constant for characterizing scale factor；Init_size is used for the initialization quantity of characterize data copy.
The third aspect, the embodiment of the invention also provides a kind of physical machines, comprising:
Configuration unit, for the corresponding sampling period to be respectively configured for different types of service；
Acquisition unit, for acquiring the target type service when reaching the target type service corresponding sampling period
The data of required transmission；
Division unit, for the data of the required transmission to be divided at least one data block；
Determination unit, for determining the first data block not being transmitted across at least one described data block, having transmitted
The second data block crossed, and determine the second data of the first data directory of first data block, second data block
Index；
Transmission unit, for sending first data block, first data directory and second data directory
To data center, so that the data center stores at least one described data block.
Preferably,
Further comprise: computing unit, for utilizing the first formula computation partition length；
First formula includes:
Wherein, k (si) for characterizing siCorresponding partition length；siI-th of the service provided for characterization platform；
period(si) for characterizing to siThe sampling period of configuration；status(si) for characterizing siState change particle size；λ
It is known constant for indicating impact factor；Defatult_size is used for the default-length of characterize data block；
The division unit is specifically used for utilizing the partition length, by the data of required transmission be divided into it is described at least
One data block；
And/or
Further comprise: storage unit, at least one data block by described in and at least one described data block
The corresponding data directory of each data block is locally stored；
And/or
The determination unit, comprising:
Determining module, for determining corresponding first check code of each data block in each data block being locally stored；
Computing module, for calculating corresponding second check code of each data block at least one described data block；
Spider module, for traversing first check code according to second check code；First verification will be located at
The corresponding data block of second check code in code will be not located in first check code as second data block
The corresponding data block of second check code is as first data block.
Fourth aspect, the embodiment of the invention also provides a kind of data centers, comprising:
Acquiring unit, for obtain the transmission of present physical machine for target type service the first data block, described the
Corresponding first data directory of one data block, corresponding second data directory of the second data block；Wherein, first data block is
The data block that the present physical machine be not transmitted across, second data block are the number that the present physical machine has been transmitted through
According to block；
Determination unit, for according to each third data block for including in the data copy that is locally stored and each
The corresponding third data directory of the third data block, and according to second data directory, determine second data
Block；
Storage unit, for storing first data block and second data block.
Preferably, further comprise: copied cells, for first data block and second data to storage
Block carries out the data copy duplication of following destination number:
Wherein, number, which is used to characterize, carries out data copy duplication to first data block and second data block
The destination number；Request_dead_lock is for characterizing multi-user's request to first data block and second number
The quantity of service block is caused according to the data contention of block；All_request is corresponding simultaneously for characterizing the target type service
Send out amount of access；A is known constant for characterizing scale factor；Init_size is used for the initialization quantity of characterize data copy.
5th aspect, the embodiment of the invention also provides a kind of data synchronous systems, comprising: in data described above
The heart and at least one physical machine described above.
The embodiment of the invention provides a kind of method of data synchronization, equipment and systems, in the number of the required transmission to acquisition
When according to being transmitted, by being divided at least one data block to the data of required transmission, by least one data block not
Once the first data block being transmitted across and its first data directory, the corresponding second data rope of the second data block having been transmitted through
Draw, be transferred to data center, without transmitting the second data block, so as to reduce the data volume in transmission process, in turn
The occupancy of network bandwidth can be reduced.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is the present invention
Some embodiments for those of ordinary skill in the art without creative efforts, can also basis
These attached drawings obtain other attached drawings.
Fig. 1 is a kind of method flow diagram provided by one embodiment of the present invention；
Fig. 2 is another method flow diagram provided by one embodiment of the present invention；
Fig. 3 is another method flow diagram provided by one embodiment of the present invention；
Fig. 4 is physical machine structural schematic diagram provided by one embodiment of the present invention；
Fig. 5 is data center's structural schematic diagram provided by one embodiment of the present invention；
Fig. 6 is data synchronous system structural schematic diagram provided by one embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments, based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without making creative work, shall fall within the protection scope of the present invention.
As shown in Figure 1, being applied to physical machine, for inhomogeneity the embodiment of the invention provides a kind of method of data synchronization
The corresponding sampling period is respectively configured in the service of type；This method may comprise steps of:
Step 101: when reaching the target type service corresponding sampling period, acquiring and passed needed for the target type service
Defeated data.
Some new datas may be generated at regular intervals in each physical machine, whenever these generate new datas it
Afterwards, it needs to store these data.Wherein, some data require real-time, some data do not want real-time
It asks, therefore, it is necessary to the real-times to data to judge.
In the present embodiment, can be respectively configured to different types of service the sampling period, to be directed to different types of clothes
Business carries out data acquisition and transmission, to realize the classification storage to service.
Step 102: the data of the required transmission are divided at least one data block.
In order to reduce the occupancy of network bandwidth, the data transmitted needed for needing are divided at least one data block.For example,
The data of required transmission are 1M, and the data of the required transmission can be divided into 10 data blocks.
Step 103: determining the first data block not being transmitted across at least one described data block, have been transmitted through
Second data block, and determine the second data directory of the first data directory of first data block, second data block.
In the present embodiment, the data block for being transferred to data center can be stored in physical machine.And pass through object
Each data block that reason machine was locally stored have been transmitted through determines which has been transmitted through at least one data block,
Which was not transmitted across.
Step 104: by first data block, first data directory and second data directory, being sent to number
According to center, so that the data center stores at least one described data block.
In order to reduce the occupancy of network bandwidth, the first data block not being transmitted across can be only transmitted, and for
The second data block being transmitted across can not have to transmission, and only corresponding second data directory of the second data block need to be transferred in data
The heart, thereby may be ensured that network occupancy compared with the existing technology reduces the network occupancy of the second data block.
According to above-described embodiment scheme, when the data of the required transmission to acquisition are transmitted, by required transmission
Data be divided at least one data block, the first data block and its first being not transmitted across at least one data block
Data directory, corresponding second data directory of the second data block having been transmitted through, are transferred to data center, without transmission
Second data block so as to reduce the data volume in transmission process, and then can reduce the occupancy of network bandwidth.
In an embodiment of the invention, the division of the data to required transmission can be realized in the following way: into one
Step includes: to utilize the first formula computation partition length；
First formula includes:
Wherein, k (si) for characterizing siCorresponding partition length；siI-th of the service provided for characterization platform；
period(si) for characterizing to siThe sampling period of configuration；status(si) for characterizing siState change particle size；λ
It is known constant for indicating impact factor；Defatult_size is used for the default-length of characterize data block；Wherein, in the clothes
In the case that the type of business determines, status (si) it is known parameter.Wherein, the setting of the defatult_size be in order to
Guarantee finally determining minimum default-length of partition length.
Wherein, which can be with byte number, for example, 10 bytes；It is also possible to specific space hold amount, example
Such as, 100KB.
The data of the required transmission are divided at least one data block, comprising: the partition length is utilized, it will be required
The data of transmission are divided at least one described data block.
For example, k (si) it is 10 bytes, the data of required transmission are 100 bytes, then can be by the required transmission
Data are divided into 10 data blocks, 10 bytes of each data block.
For another example, k (si) it is 10 bytes, the data of required transmission are 99 bytes, then can be by the required transmission
Data are divided into 10 data blocks, 10 bytes of each data block in preceding 9 data blocks, and last 1 data block is 9 words
Section.
In an embodiment of the invention, in first data block, first data directory and the second data rope
After data center is given in initiation, when in order to guarantee to carry out transmission of data blocks next time, it is possible to reduce the data number of blocks of transmission,
It can be sent to after data center by first data block, first data directory and second data directory,
Further comprise: by the corresponding number of each data block at least one described data block and at least one described data block
It is locally stored according to index.
In an embodiment of the invention, it can be determined as follows at least one described data block and not transmit
The first data block crossed, the second data block having been transmitted through, comprising:
Determine corresponding first check code of each data block in each data block being locally stored；
Calculate corresponding second check code of each data block at least one described data block；
According to second check code, first check code is traversed；Described the in first check code will be located at
The corresponding data block of two check codes is as second data block, second verification that will be not located in first check code
The corresponding data block of code is as first data block.
Wherein, the calculation of corresponding first check code of each data block in each data block being locally stored, with
The calculation of corresponding second check code of each data block is identical at least one data block.
For example, at least one data block includes 10 data blocks, it is respectively as follows: data block 1, data block 2, data block
3 ..., data block 10, corresponding second check code of each data block be respectively as follows: A1, A2, A3 ..., A10.Assuming that
It include the check code of A1, A2 and A3 in the first check code being locally stored, then data block 1, data block 2 and data can be determined
Block 3 is the data block having been transmitted through, data block 4, data block 5, data block 6 ... data block 10 is the number not being transmitted across
According to block.
In an embodiment of the invention, each data at least one data block can be calculated in the following way
Corresponding second check code of block, the technical approach may include:
Each data block pair at least one described data block is calculated using the second formula, third formula and the 4th formula
The second check code answered；
Second formula includes:
The third formula includes:
4th formula includes:
Adler32 (1, k)=A (1, k)+216B(1,k)；
Wherein, Adler32 (1, k) is for characterizing the second check code；A (1, k) is for characterizing the first intermediate parameters, B (1, k)
For characterizing the second intermediate parameters；Data [j] is used to characterize the corresponding data of j-th of byte in current data block；M is used for table
Levy known constant；K is for characterizing the byte number that the current data block includes.
In the second check code being calculated according to above-mentioned calculation, if be calculated according to above-mentioned calculation
Two check codes are all different with the first check code being locally stored, then show different second check code corresponding second
Data block is not transmitted across centainly；If identical as the first check code being locally stored, show identical second check code pair
The second data block answered may have been transmitted through, it is also necessary to unique check value is utilized, for example, MD5 value, is further verified,
If identical using the further check results of MD5 value, show that corresponding second data block of second check code has been transmitted through, it is no
Then, it is not transmitted across.
In an embodiment of the invention, can also by directly by unique check value in a manner of calculate the second check code, with
Determine the first data block and the second data block.
Referring to FIG. 2, being applied to data center the embodiment of the invention also provides a kind of method of data synchronization, comprising:
Step 201: obtaining the first data block for target type service, first data that present physical machine is sent
Corresponding first data directory of block, corresponding second data directory of the second data block；Wherein, first data block is described works as
The data block that preceding physical machine be not transmitted across, second data block are the data block that the present physical machine has been transmitted through.
Step 202: according to each third data block for including in the data copy being locally stored and each described
The corresponding third data directory of three data blocks, and according to second data directory, determine second data block.
The data copy of each data block is locally stored in data center, wherein data copy is a kind of raising data visit
Ask the general technology of efficiency, Fault Tolerance, load balance ability.It not only include each third data block in data copy,
It further include the corresponding third data directory of each third data block, by being searched and the second data rope in third data directory
Draw identical data directory, and determine corresponding data block using the identical data directory, which is
The second data block having been transmitted through.
Step 203: storing first data block and second data block.
In the present embodiment, service S can also be setiThe threshold value of corresponding data volume, for example, the threshold value is 100, at this
When the data block number that service includes reaches 100, by the corresponding data block storage of the service into cloud storage.
Further, data center is arrived the data storage for needing real-time by judging demand of the data to real-time
In real-time data base, the data for not needing real-time are stored into common database.
In an embodiment of the invention, unreasonable due to the number of data copy, it is likely to result in the storage of platform
Pressure wastes a large amount of memory space, therefore, can be determined as follows the number of data copy, further comprise: is right
First data block and second data block of storage carry out the data copy duplication of following destination number:
Wherein, number, which is used to characterize, carries out data copy duplication to first data block and second data block
The destination number；Request_dead_lock is for characterizing multi-user's request to first data block and second number
The quantity of service block is caused according to the data contention of block；All_request is corresponding simultaneously for characterizing the target type service
Send out amount of access；A is known constant for characterizing scale factor；Init_size is used for the initialization quantity of characterize data copy.
Below by taking the platform for providing service is PAAS platform as an example, which includes at least a physical machine and data
Center is interacted between physical machine and data center by wherein one, detailed to carry out to the data synchronization process of realization
Illustrate, referring to FIG. 3, this method may comprise steps of:
Step 301: the corresponding sampling period is respectively configured for different types of service in physical machine.
Some new datas may be generated at regular intervals in each physical machine, whenever these generate new datas it
Afterwards, it needs to store these data.
Assuming that including 4 kinds of service types, these four service types are carried out with configuration as shown in table 1 below:
Table 1:
| Service type | Sampling period/time (s) |
| Class1 | 60 |
| Type 2 | 180 |
| Type 3 | 300 |
| Type 4 | 90 |
In the present embodiment, different acquisition units can be utilized respectively, respectively for aforementioned four type service into
The acquisition of row data, each acquisition unit, according to the sampling period configured in above-mentioned table 1, whenever the arrival corresponding sampling period
When, execute the acquisition to the respective type service data.
In the present embodiment, user also needs to register monitoring information to the Agent of data center, such as monitor control index, monitoring
Object, monitor mode etc..
In order to provide preferable scalability, the Agent in physical machine provides the mode of two kinds of extensions: the first is logical
The mode for crossing script calling transmits monitoring script to Agent by user, calls script to run monitoring module, acquisition clothes by Agent
Business status information.The second way is the API by calling Agent, is extended equally to it and can achieve acquisition service shape
The purpose of state information.
Step 302: when reaching the target type service corresponding sampling period, acquiring and passed needed for the target type service
Defeated data.
In the present embodiment, it may include more virtual machines in physical machine, Agent can be configured on each virtual machine
Acquisition and processing to data may be implemented in agency, Agent agency.
Step 303: according to service type computation partition length.
In the present embodiment, following formula (1) computation partition length can be passed through:
Wherein, k (si) for characterizing siCorresponding partition length；siI-th of the service provided for characterization platform；
period(si) for characterizing to siThe sampling period of configuration；status(si) for characterizing siState change particle size；λ
It is known constant for indicating impact factor；Defatult_size is used for the default-length of characterize data block.
Wherein, in the case where the type of the service determines, status (si) it is known parameter.
Wherein, the setting of the defatult_size is to guarantee that the minimum default of the partition length finally determined is long
Degree.For example, the smallest default length is 5 bytes.
Step 304: using the partition length calculated, the data of required transmission being divided into m data block.
Assuming that the m data block that the data transmitted needed for target type is serviced mark off is as follows:Wherein,
Step 305: obtaining each data block being locally stored.
The each data block and each data block pair of data center are had transmitted in being locally stored for physical machine
The data directory answered.
Step 306: determining corresponding first check code of each data block in each data block being locally stored.
Step 307: using the calculation of the first check code, calculating each data block in m data block corresponding the
Two check codes, and the first data block that do not transmitted in m data block and is determined according to the first check code and the second check code
The second data block through transmitting.
In the present embodiment, it in order to guarantee the consistency of check code, needs to count using the calculation of the first check code
Calculate the second check code.
Wherein, which can be the unique check value for directly calculating each data block, which can
Think MD5 value.For example, corresponding second check code of m data block are as follows: A1, A2, A3 ..., Am.What is be locally stored
It include: A1, A2 and A3 in first check code, then it is determined that data block 1, data block 2 and data block 3 are the data having been transmitted through
Block, by data block 1, data block 2 and data block 3 be used as the second data block, by data block 4, data block 5, data block 6 ..., number
According to block m as the first data block not being transmitted across.
Since the calculation of unique check value is complex, the used time is longer, therefore, can calculate in the following way
Two check codes:
In an embodiment of the invention, each data at least one data block can be calculated in the following way
Corresponding second check code of block, the calculation may include:
Corresponding second verification of each data block in m data block is calculated using formula (2), formula (3) and formula (4)
Code；
Adler32 (1, k)=A (1, k)+216B(1,k) (4)
Wherein, Adler32 (1, k) is for characterizing the second check code；A (1, k) is for characterizing the first intermediate parameters, B (1, k)
For characterizing the second intermediate parameters；Data [j] is used to characterize the corresponding data of j-th of byte in current data block；M is used for table
Levy known constant；K is for characterizing the byte number that the current data block includes.
In the second check code being calculated according to above-mentioned calculation, if be calculated according to above-mentioned calculation
Second check code is all different with the first check code being locally stored, then shows different second check code corresponding
Two data blocks are not transmitted across centainly；Assuming that corresponding second check code of m data block be respectively as follows: A1, A2, A3 ..., Am.
A1, A2, A3 are respectively positioned in the first check code being locally stored, then, A4, A5, A6 ..., Am be not transmitted across, can be by number
According to block 4, data block 5, data block 6 ..., data block m be determined as the second data block.
If identical as the first check code being locally stored, show corresponding second data block of identical second check code
It may have been transmitted through, it is also necessary to unique check value is utilized, for example, MD5 value, is further verified.To data block 1, data
Block 2 and the needs of data block 3 are further verified, and data block 1, data block 2 and the corresponding MD5 value of data block 3 are calculated, with
And the corresponding MD5 value of three data blocks that the first check code being locally stored is A1, A2 and A3 is calculated, if corresponding data
The MD5 value of block is identical, then shows that the data block has been transmitted through；Otherwise, which was not transmitted across.For example, A1, A2 and A3
It has been transmitted through, data block 1, data block 2 and data block 3 is determined as the first data block.
Step 308: determining corresponding first data directory of the first data block, and determine the second data block corresponding second
Data directory.
Wherein, first data directory needs are generated according to the first data block.
Second data directory can be determined according to the index being locally stored.
Step 309: by the first data block, the first data directory and the second data directory, being sent to data center, and by
One data block and the first data directory are locally stored.
In order to reduce the occupancy of network bandwidth, the first data block not being transmitted across can be only transmitted, and for
The second data block being transmitted across can not have to transmission, and only corresponding second data directory of the second data block need to be transferred in data
The heart, thereby may be ensured that network occupancy compared with the existing technology reduces the network occupancy of the second data block.
In the present embodiment, data center is sent in first data block, the first data directory and the second data directory
When later, in order to guarantee to carry out transmission of data blocks next time, it is possible to reduce the data number of blocks of transmission may further include:
The corresponding data directory of each data block in an at least data block and at least one data block is locally stored.
Wherein, the data of storage can also include the first data block and the corresponding timestamp of the second data block.
In the present embodiment, the first data block, the first data directory and the second data directory can also be compressed, and
Compressed data packet is sent to data center.
Step 310: data center using include in the data copy of storage each third data block, each third number
According to the corresponding third data directory of block, and according to the second data directory, determine the second data block.
Since the second data block has been transmitted across to data center, it is stored with the second data block in the data center
With the second data directory.
Wherein it is possible to obtain the second data block by data copy.
The data copy of each data block is locally stored in data center, wherein data copy is a kind of raising data visit
Ask the general technology of efficiency, Fault Tolerance, load balance ability.It not only include each third data block in data copy,
It further include the corresponding third data directory of each third data block, by being searched and the second data rope in third data directory
Draw identical data directory, and determine corresponding data block using the identical data directory, which is
The second data block having been transmitted through.
Step 311: according to threshold value corresponding to the service of the target type of setting, determining first data block and second
Whether the number of data block is greater than the threshold value, if more than, then by first data block and the storage of the second data block into cloud storage,
If being not more than, by first data block and the storage of the second data block into real-time data base.
Further, service S can also be setiThe threshold value of corresponding data volume, for example, the threshold value is 100, in the service
Including data block number reach 100 when, by the service corresponding data block storage into cloud storage.
Step 312: data copy duplication is carried out to the first data block and the second data block.
The number of data copy it is unreasonable, be likely to result in the storage pressure of platform, waste a large amount of memory space, because
This, can be determined as follows the number of data copy:
The quantity of the data copy can be determined by following formula (5):
Wherein, number, which is used to characterize, carries out data copy duplication to first data block and second data block
The destination number；Request_dead_lock is for characterizing multi-user's request to first data block and second number
The quantity of service block is caused according to the data contention of block；All_request is corresponding simultaneously for characterizing the target type service
Send out amount of access；A is known constant for characterizing scale factor；Init_size is used for the initialization quantity of characterize data copy.
Referring to FIG. 4, may include: the embodiment of the invention also provides a kind of physical machine
Configuration unit 401, for the corresponding sampling period to be respectively configured for different types of service；
Acquisition unit 402, for when reaching the target type service corresponding sampling period, acquiring the target type clothes
The data transmitted needed for business；
Division unit 403, for the data of the required transmission to be divided at least one data block；
Determination unit 404, for determining the first data block not being transmitted across at least one described data block, having passed
Second data block of defeated mistake, and determine the second number of the first data directory of first data block, second data block
According to index；
Transmission unit 405 is used for first data block, first data directory and second data directory,
It is sent to data center, so that the data center stores at least one described data block.
In an embodiment of the invention, it may further include: computing unit, for utilizing the first formula computation partition
Length；
First formula includes:
Wherein, k (si) for characterizing siCorresponding partition length；siI-th of the service provided for characterization platform；
period(si) for characterizing to siThe sampling period of configuration；status(si) for characterizing siState change particle size；λ
It is known constant for indicating impact factor；Defatult_size is used for the default-length of characterize data block；
The division unit is specifically used for utilizing the partition length, by the data of required transmission be divided into it is described at least
One data block；
In an embodiment of the invention, it may further include: storage unit, at least one data by described in
The corresponding data directory of each data block is locally stored in block and at least one described data block；
In an embodiment of the invention, the determination unit, comprising:
Determining module, for determining corresponding first check code of each data block in each data block being locally stored；
Computing module, for calculating corresponding second check code of each data block at least one described data block；
Spider module, for traversing first check code according to second check code；First verification will be located at
The corresponding data block of second check code in code will be not located in first check code as second data block
The corresponding data block of second check code is as first data block.
Referring to FIG. 5, may include: the embodiment of the invention also provides a kind of data center
Acquiring unit 501, for obtaining the first data block, described for target type service of present physical machine transmission
Corresponding first data directory of first data block, corresponding second data directory of the second data block；Wherein, first data block
For the data block that the present physical machine be not transmitted across, the second data block present physical machine is had been transmitted through
Data block；
Determination unit 502, for according to each third data block for including in the data copy that is locally stored and each
The corresponding third data directory of a third data block, and according to second data directory, determine second number
According to block；
Storage unit 503, for storing first data block and second data block.
In an embodiment of the invention, it may further include: copied cells, for first data to storage
Block and second data block carry out the data copy duplication of following destination number:
Wherein, number, which is used to characterize, carries out data copy duplication to first data block and second data block
The destination number；Request_dead_lock is for characterizing multi-user's request to first data block and second number
The quantity of service block is caused according to the data contention of block；All_request is corresponding simultaneously for characterizing the target type service
Send out amount of access；A is known constant for characterizing scale factor；Init_size is used for the initialization quantity of characterize data copy.
Referring to FIG. 6, may include: any of the above-described described the embodiment of the invention also provides a kind of data synchronous system
Data center 50 and at least one any of the above-described described physical machine 40.
In conclusion each embodiment of the present invention at least may be implemented it is following the utility model has the advantages that
1, in embodiments of the present invention, when the data of the required transmission to acquisition are transmitted, by required transmission
Data be divided at least one data block, the first data block and its first being not transmitted across at least one data block
Data directory, corresponding second data directory of the second data block having been transmitted through, are transferred to data center, without transmission
Second data block so as to reduce the data volume in transmission process, and then can reduce the occupancy of network bandwidth.
2, in embodiments of the present invention, the Agent agency in physical machine can provide flexible extension function, and user can be with
By way of Interface Expanding or script language calling, the data monitoring acquisition of Agent agency is dynamically extended.With
Family does not need to be modified entire plateform system because customizing service is introduced.
3, in embodiments of the present invention, service-oriented more copy data storage strategies are guaranteeing the visit of data real-time storage
On the basis of asking, the redundancy quantity of data copy is not only reduced, and improves the concurrently access degree of system.
The contents such as the information exchange between each unit, implementation procedure in above-mentioned apparatus, due to implementing with the method for the present invention
Example is based on same design, and for details, please refer to the description in the embodiment of the method for the present invention, and details are not described herein again.
It should be noted that, in this document, such as first and second etc relational terms are used merely to an entity
Or operation is distinguished with another entity or operation, is existed without necessarily requiring or implying between these entities or operation
Any actual relationship or order.Moreover, the terms "include", "comprise" or its any other variant be intended to it is non-
It is exclusive to include, so that the process, method, article or equipment for including a series of elements not only includes those elements,
It but also including other elements that are not explicitly listed, or further include for this process, method, article or equipment institute
Intrinsic element.In the absence of more restrictions, the element limited by sentence " including a 〃 ", not
There is also other identical factors in the process, method, article or apparatus that includes the element for exclusion.
Those of ordinary skill in the art will appreciate that: realize that all or part of the steps of above method embodiment can pass through
The relevant hardware of program instruction is completed, and program above-mentioned can store in computer-readable storage medium, the program
When being executed, step including the steps of the foregoing method embodiments is executed；And storage medium above-mentioned includes: ROM, RAM, magnetic disk or light
In the various media that can store program code such as disk.
Finally, it should be noted that the foregoing is merely presently preferred embodiments of the present invention, it is merely to illustrate skill of the invention
Art scheme, is not intended to limit the scope of the present invention.Any modification for being made all within the spirits and principles of the present invention,
Equivalent replacement, improvement etc., are included within the scope of protection of the present invention.
Claims (8)
1. a kind of method of data synchronization, which is characterized in that be applied to physical machine, be respectively configured accordingly for different types of service
Sampling period；The described method includes:
When reaching the target type service corresponding sampling period, the data transmitted needed for the target type service are acquired；
The data of the required transmission are divided at least one data block；
Determine the first data block not being transmitted across at least one described data block, the second data block having been transmitted through, with
And determine the second data directory of the first data directory of first data block, second data block；
By first data block, first data directory and second data directory, it is sent to data center, so that institute
It states data center and stores at least one described data block；
Before the data of the required transmission are divided at least one data block, further comprise: utilizing the first formula meter
Calculate partition length；
First formula includes:
Wherein, k (si) for characterizing SiCorresponding partition length；SiI-th of the service provided for characterization platform；period(si)
For characterizing to SiThe sampling period of configuration；status(si) for characterizing SiState change particle size；λ is for indicating shadow
The factor is rung, is known constant；Defatult_size is used for the default-length of characterize data block；
The data of the required transmission are divided at least one data block, comprising: the partition length is utilized, by required transmission
Data be divided at least one described data block；
By first data block, first data directory and second data directory, it is sent to after data center,
Further comprise: by the corresponding number of each data block at least one described data block and at least one described data block
It is locally stored according to index；
The first data block not being transmitted across at least one data block described in the determination, the second data having been transmitted through
Block, comprising:
Determine corresponding first check code of each data block in each data block being locally stored；
Calculate corresponding second check code of each data block at least one described data block；
According to second check code, first check code is traversed；Second school in first check code will be located at
The corresponding data block of code is tested as second data block, second check code pair that will be not located in first check code
The data block answered is as first data block.
2. the method according to claim 1, wherein each number at least one data block described in the calculating
According to corresponding second check code of block, comprising:
It is corresponding that each data block at least one described data block is calculated using the second formula, third formula and the 4th formula
Second check code；
Second formula includes:
The third formula includes:
4th formula includes:
Adler32 (1, k)=A (1, k)+216B(1,k)；
Wherein, Adler32 (1, k) is for characterizing the second check code；A (1, k) is used for for characterizing the first intermediate parameters, B (1, k)
Characterize the second intermediate parameters；Data [j] is used to characterize the corresponding data of j-th of byte in current data block；M is for characterizing
Know constant；K is for characterizing the byte number that the current data block includes.
3. a kind of method of data synchronization, which is characterized in that be applied to data center, comprising:
Obtain the first data block for target type service, first data block corresponding first that present physical machine is sent
Corresponding second data directory of data directory, the second data block；Wherein, first data block be the present physical machine not
The data block being transmitted across, second data block are the data block that the present physical machine has been transmitted through；
It is corresponding according to each third data block for including in the data copy being locally stored and each described third data block
Third data directory, and according to second data directory, determine second data block；
Store first data block and second data block.
4. according to the method described in claim 3, it is characterized in that, further comprising: to first data block of storage and
Second data block carries out the data copy duplication of following destination number:
Wherein, number carries out described in data copy duplication first data block and second data block for characterizing
Destination number；Request_dead_lock is for characterizing multi-user's request to first data block and second data block
Data contention cause the quantity of service block；For characterizing, the target type service is corresponding concurrently to visit all_request
The amount of asking；A is known constant for characterizing scale factor；Init_size is used for the initialization quantity of characterize data copy.
5. a kind of physical machine characterized by comprising
Configuration unit, for the corresponding sampling period to be respectively configured for different types of service；
Acquisition unit, for when reaching the target type service corresponding sampling period, acquiring needed for the target type service
The data of transmission；
Division unit, for the data of the required transmission to be divided at least one data block；
Determination unit, for determining the first data block not being transmitted across at least one described data block, having been transmitted through
Second data block, and determine the second data directory of the first data directory of first data block, second data block；
Transmission unit, for being sent to number for first data block, first data directory and second data directory
According to center, so that the data center stores at least one described data block；
Further comprise: computing unit, for utilizing the first formula computation partition length；
First formula includes:
Wherein, k (si) for characterizing SiCorresponding partition length；SiI-th of the service provided for characterization platform；period(si)
For characterizing to SiThe sampling period of configuration；status(si) for characterizing SiState change particle size；λ is for indicating shadow
The factor is rung, is known constant；Defatult_size is used for the default-length of characterize data block；
The division unit is specifically used for utilizing the partition length, by the data of required transmission be divided into it is described at least one
Data block；
Further comprise: storage unit, for each at least one data block by described in and at least one described data block
The corresponding data directory of a data block is locally stored；
The determination unit, comprising:
Determining module, for determining corresponding first check code of each data block in each data block being locally stored；
Computing module, for calculating corresponding second check code of each data block at least one described data block；
Spider module, for traversing first check code according to second check code；It will be located in first check code
The corresponding data block of second check code as second data block, will be not located in first check code described in
The corresponding data block of second check code is as first data block.
6. a kind of data center characterized by comprising
Acquiring unit, for obtaining the first data block for target type service, first number that present physical machine is sent
According to corresponding first data directory of block, corresponding second data directory of the second data block；Wherein, first data block is described
The data block that present physical machine be not transmitted across, second data block are the data that the present physical machine has been transmitted through
Block；
Determination unit, for according to each third data block for including in the data copy being locally stored and each
The corresponding third data directory of third data block, and according to second data directory, determine second data block；
Storage unit, for storing first data block and second data block.
7. data center according to claim 6, which is characterized in that further comprise: copied cells, for storage
First data block and second data block carry out the data copy duplication of following destination number:
Wherein, number carries out described in data copy duplication first data block and second data block for characterizing
Destination number；Request_dead_lock is for characterizing multi-user's request to first data block and second data block
Data contention cause the quantity of service block；For characterizing, the target type service is corresponding concurrently to visit all_request
The amount of asking；A is known constant for characterizing scale factor；Init_size is used for the initialization quantity of characterize data copy.
8. a kind of data synchronous system characterized by comprising data center described in the claims 6 or 7 and at least
Physical machine described in one the claims 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610451188.0A CN106209974B (en) | 2016-06-21 | 2016-06-21 | A kind of method of data synchronization, equipment and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610451188.0A CN106209974B (en) | 2016-06-21 | 2016-06-21 | A kind of method of data synchronization, equipment and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106209974A CN106209974A (en) | 2016-12-07 |
| CN106209974B true CN106209974B (en) | 2019-03-12 |
Family
ID=57460677
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610451188.0A Active CN106209974B (en) | 2016-06-21 | 2016-06-21 | A kind of method of data synchronization, equipment and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106209974B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107070740A (en) * | 2017-03-11 | 2017-08-18 | 郑州云海信息技术有限公司 | A kind of efficient PAAS platform monitoring methods and system |
| CN107357746B (en) * | 2017-07-26 | 2020-05-29 | 郑州云海信息技术有限公司 | Communication method and system |
| CN107241447B (en) * | 2017-07-31 | 2019-09-13 | Oppo广东移动通信有限公司 | Data synchronize management-control method, device, storage medium and electronic equipment |
| CN108418746B (en) * | 2018-02-13 | 2020-06-12 | 论客科技（广州）有限公司 | Mail synchronization method and device and computer readable storage medium |
| CN109733444B (en) * | 2018-09-19 | 2020-05-19 | 比亚迪股份有限公司 | Database system and train monitoring management equipment |
| DE102020105794B4 (en) * | 2020-03-04 | 2022-02-03 | Infineon Technologies Ag | Device, controller for a device and method for communication |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101355588B (en) * | 2008-09-08 | 2012-08-01 | 创新科存储技术（深圳）有限公司 | Data transmission method and transmission terminal base on peer-to-peer network |
| CN102436478B (en) * | 2011-10-12 | 2013-06-19 | 浪潮(北京)电子信息产业有限公司 | System and method for accessing massive data |
| CN103873503A (en) * | 2012-12-12 | 2014-06-18 | 鸿富锦精密工业（深圳）有限公司 | Data block backup system and method |
| CN104063377A (en) * | 2013-03-18 | 2014-09-24 | 联想(北京)有限公司 | Information processing method and electronic equipment using same |
| CN104348884A (en) * | 2013-08-08 | 2015-02-11 | 中国科学院计算机网络信息中心 | Cloud storage automatic synchronization method |
-
2016
- 2016-06-21 CN CN201610451188.0A patent/CN106209974B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101355588B (en) * | 2008-09-08 | 2012-08-01 | 创新科存储技术（深圳）有限公司 | Data transmission method and transmission terminal base on peer-to-peer network |
| CN102436478B (en) * | 2011-10-12 | 2013-06-19 | 浪潮(北京)电子信息产业有限公司 | System and method for accessing massive data |
| CN103873503A (en) * | 2012-12-12 | 2014-06-18 | 鸿富锦精密工业（深圳）有限公司 | Data block backup system and method |
| CN104063377A (en) * | 2013-03-18 | 2014-09-24 | 联想(北京)有限公司 | Information processing method and electronic equipment using same |
| CN104348884A (en) * | 2013-08-08 | 2015-02-11 | 中国科学院计算机网络信息中心 | Cloud storage automatic synchronization method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106209974A (en) | 2016-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106209974B (en) | A kind of method of data synchronization, equipment and system | |
| US10237295B2 (en) | Automated event ID field analysis on heterogeneous logs | |
| CN104252481B (en) | The dynamic check method and apparatus of master-slave database consistency | |
| CN109347675A (en) | Server configuration method, device and electronic equipment | |
| CN105933408B (en) | A kind of implementation method and device of Redis universal middleware | |
| CN107534570A (en) | Virtualize network function monitoring | |
| CN109213792A (en) | Method, server-side, client, device and the readable storage medium storing program for executing of data processing | |
| CN107133231B (en) | Data acquisition method and device | |
| CN111181800B (en) | Test data processing method and device, electronic equipment and storage medium | |
| CN108347347A (en) | A kind of information processing method, device and system | |
| CN107181637A (en) | A kind of heartbeat message sending method, device and heartbeat sending node | |
| CN102025753B (en) | Load balancing method and equipment for data resources of servers | |
| CN109254914A (en) | Software development kit test method, system, computer installation and readable storage medium storing program for executing | |
| CN109783151A (en) | The method and apparatus of rule change | |
| CN106384399A (en) | IOT (Internet of Things)-office-oriented network attendance method | |
| WO2022267769A1 (en) | Method and apparatus for generating graph data | |
| CN108833592A (en) | Cloud host schedules device optimization method, device, equipment and storage medium | |
| CN109245915B (en) | Method and system for realizing server set balanced allocation | |
| CN110380890A (en) | A kind of CDN system service quality detection method and system | |
| CN110334147A (en) | A kind of method of data synchronization and device | |
| CN114598680B (en) | Domain name management method, device and storage medium | |
| RU2532714C2 (en) | Method of acquiring data when evaluating network resources and apparatus therefor | |
| CN106844233A (en) | A kind of router service data caching method and system based on Hash table | |
| US20160342899A1 (en) | Collaborative filtering in directed graph | |
| US20160253391A1 (en) | Data communication in a distributed data grid |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200522 Address after: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Patentee after: Tidal Cloud Information Technology Co.,Ltd. Address before: 250100 Ji'nan high tech Zone, Shandong, No. 1036 wave road Patentee before: INSPUR ELECTRONIC INFORMATION INDUSTRY Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Patentee after: Inspur cloud Information Technology Co., Ltd Address before: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Patentee before: Tidal Cloud Information Technology Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder |