CN106209974A - A kind of method of data synchronization, equipment and system - Google Patents

Abstract

The present invention provides a data synchronization method, device and system. The method includes: respectively configuring corresponding sampling periods for different types of services; data; divide the data to be transmitted into at least one data block; determine the first data block that has never been transmitted in the at least one data block, the second data block that has been transmitted, and determine the first data block The first data index of the block, the second data index of the second data block; sending the first data block, the first data index and the second data index to the data center, so that the The data center stores the at least one data block. According to the solution, the amount of data in the transmission process can be reduced, thereby reducing the occupation of network bandwidth.

Description

A data synchronization method, device and system

technical field

The present invention relates to the technical field of cloud computing, in particular to a data synchronization method, device and system.

Background technique

Cloud computing technology uses virtualization technology to pool computing, storage, network and other resources, and provides shared software and hardware to users in the form of services through the Internet. Among them, PAAS (Platform as a Service, Platform as a Service) platform, as a service type of cloud computing, provides software deployment and operation and maintenance as a service to software development users on demand, and has become a very popular research direction in recent years.

In the PAAS platform, in order to realize system services and various customized services, the data generated on each physical machine needs to be stored synchronously. When synchronizing the data generated on each physical machine in real time, the physical machine collects the data generated by itself, sends the data to the data center through network resources, and the data center stores the data in the real-time database.

Network resources are extremely precious to the PAAS platform, and in the prior art, a large amount of data will be generated on each physical machine, resulting in a large occupation of network bandwidth.

Contents of the invention

The embodiment of the present invention provides a data synchronization method, device and system, so as to reduce the occupation of network bandwidth.

In the first aspect, the embodiment of the present invention provides a data synchronization method, which is applied to a physical machine and configures corresponding sampling periods for different types of services; the method includes:

When the sampling period corresponding to the target type service is reached, collect the data that the target type service needs to transmit;

dividing the data to be transmitted into at least one data block;

Determining the first data block that has not been transmitted and the second data block that has been transmitted in the at least one data block, and determining the first data index of the first data block and the second data of the second data block index;

sending the first data block, the first data index, and the second data index to a data center, so that the data center stores the at least one data block.

Preferably,

Before dividing the data to be transmitted into at least one data block, it further includes: using a first formula to calculate the division length;

The first formula includes:

$\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; z</span>}\mathrm{<span\; class="notranslate">\; e</span>}$

Among them, k(S _i ) is used to represent the division length corresponding to S _i ; S _i is used to represent the i-th service provided by the platform; period(s _i ) is used to represent the sampling period configured for S _i ; status(s _i ) is used to characterize the state change granularity evaluation of s _i ; λ is used to represent the impact factor, which is a known constant; default_size is used to represent the default length of the data block;

Dividing the data to be transmitted into at least one data block includes: dividing the data to be transmitted into the at least one data block by using the division length;

and / or,

After sending the first data block, the first data index, and the second data index to the data center, further comprising: sending the at least one data block, and each of the at least one data block The data index corresponding to the data block is stored locally;

and / or,

The determination of the first data block that has never been transmitted and the second data block that has been transmitted in the at least one data block includes:

Determine the first check code corresponding to each data block in each data block stored locally;

calculating a second check code corresponding to each data block in the at least one data block;

Traverse the first check code according to the second check code; use the data block corresponding to the second check code in the first check code as the second data block, and A data block corresponding to the second check code in the first check code is used as the first data block.

Preferably, the calculating the second check code corresponding to each data block in the at least one data block includes:

calculating a second check code corresponding to each data block in the at least one data block by using the second formula, the third formula and the fourth formula;

The second formula includes:

$\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; mod</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ;</span>$

The third formula includes:

$\mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; mod</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ;</span>$

The fourth formula includes:

Adler32(1,k)=A(1,k)+2 ¹⁶ B(1,k);

Among them, Adler32(1,k) is used to represent the second check code; A(1,k) is used to represent the first intermediate parameter, B(1,k) is used to represent the second intermediate parameter; data[j] is used to represent is used to represent the data corresponding to the jth byte in the current data block; M is used to represent a known constant; k is used to represent the number of bytes included in the current data block.

In the second aspect, the embodiment of the present invention also provides a data synchronization method applied to a data center, including:

Obtain the first data block for the target type service sent by the current physical machine, the first data index corresponding to the first data block, and the second data index corresponding to the second data block; wherein, the first data block is the A data block that has never been transmitted by the current physical machine, and the second data block is a data block that has been transmitted by the current physical machine;

determining the second data block according to each third data block included in the locally stored data copy, and a third data index corresponding to each of the third data blocks, and according to the second data index;

The first data block and the second data block are stored.

Preferably, it further includes: performing the following target number of data copies on the stored first data block and the second data block:

Among them, number is used to represent the target number of data copy replication for the first data block and the second data block; request_dead_lock is used to represent the multi-user request for the first data block and the second data block The number of service blocks caused by block data competition; all_request is used to represent the concurrent access amount corresponding to the target type of service; a is used to represent the scaling factor, which is a known constant; init_size is used to represent the initialization number of data copies.

In a third aspect, the embodiment of the present invention also provides a physical machine, including:

The configuration unit is configured to respectively configure corresponding sampling periods for different types of services;

A collection unit, configured to collect the data required to be transmitted by the target type service when the sampling period corresponding to the target type service is reached;

a division unit, configured to divide the data to be transmitted into at least one data block;

A determining unit, configured to determine a first data block that has not been transmitted in the at least one data block, a second data block that has been transmitted, and determine the first data index of the first data block, the second data block the second data index of the block;

A sending unit, configured to send the first data block, the first data index, and the second data index to a data center, so that the data center stores the at least one data block.

Preferably,

Further comprising: a calculation unit, configured to calculate the division length by using the first formula;

The first formula includes:

$\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; z</span>}\mathrm{<span\; class="notranslate">\; e</span>}$

Among them, k(s _i ) is used to represent the division length corresponding to s _i ; s _i is used to represent the i-th service provided by the platform; period(s _i ) is used to represent the sampling period configured for s _i ; status(s _i ) is used to characterize the state change granularity evaluation of s _i ; λ is used to represent the impact factor, which is a known constant; default_size is used to represent the default length of the data block;

The division unit is specifically configured to divide the data to be transmitted into the at least one data block by using the division length;

and / or,

Further comprising: a storage unit, configured to locally store the at least one data block and a data index corresponding to each data block in the at least one data block;

and / or,

The determination unit includes:

A determining module, configured to determine a first check code corresponding to each data block in each data block stored locally;

A calculation module, configured to calculate a second check code corresponding to each data block in the at least one data block;

A traversal module, configured to traverse the first check code according to the second check code; use the data block corresponding to the second check code in the first check code as the second check code For a data block, a data block corresponding to the second check code that is not located in the first check code is used as the first data block.

In a fourth aspect, the embodiment of the present invention also provides a data center, including:

The acquiring unit is configured to acquire the first data block sent by the current physical machine for the service of the target type, the first data index corresponding to the first data block, and the second data index corresponding to the second data block; wherein, the first A data block is a data block that has not been transmitted by the current physical machine, and the second data block is a data block that has been transmitted by the current physical machine;

A determining unit, configured to determine the first data block according to each third data block included in the locally stored data copy, and the third data index corresponding to each third data block, and according to the second data index Two data blocks;

A storage unit, configured to store the first data block and the second data block.

Preferably, it further includes: a copying unit, configured to perform the following target number of data copies on the stored first data block and the second data block:

Among them, number is used to represent the target number of data copy replication for the first data block and the second data block; request_dead_lock is used to represent the multi-user request for the first data block and the second data block The number of service blocks caused by block data competition; all_request is used to represent the concurrent access amount corresponding to the target type of service; a is used to represent the scaling factor, which is a known constant; init_size is used to represent the initialization number of data copies.

In a fifth aspect, an embodiment of the present invention further provides a data synchronization system, including: the above-mentioned data center, and at least one of the above-mentioned physical machines.

Embodiments of the present invention provide a data synchronization method, device, and system. When transmitting the collected data to be transmitted, the data to be transmitted is divided into at least one data block, and the data in the at least one data block is The first data block that has never been transmitted and its first data index, and the second data index corresponding to the second data block that has been transmitted can be transmitted to the data center, without the need to transmit the second data block, which can reduce the transmission process. The amount of data, which in turn can reduce the occupation of network bandwidth.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a kind of method flowchart provided by one embodiment of the present invention;

Fig. 2 is another method flowchart provided by an embodiment of the present invention;

Fig. 3 is another method flowchart provided by an embodiment of the present invention;

Fig. 4 is a schematic structural diagram of a physical machine provided by an embodiment of the present invention;

Fig. 5 is a schematic structural diagram of a data center provided by an embodiment of the present invention;

Fig. 6 is a schematic structural diagram of a data synchronization system provided by an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection of the present invention. scope.

As shown in Figure 1, the embodiment of the present invention provides a data synchronization method, which is applied to a physical machine and configures corresponding sampling periods for different types of services; the method may include the following steps:

Step 101: When the sampling period corresponding to the target type service is reached, collect the data required to be transmitted by the target type service.

Some new data may be generated at regular intervals on each physical machine, and these data need to be stored whenever new data is generated. Among them, some data have requirements for real-time performance, and some data have no requirement for real-time performance. Therefore, it is necessary to judge the real-time performance of the data.

In this embodiment, sampling periods may be configured for different types of services, so as to collect and transmit data for different types of services, thereby implementing classified storage of services.

Step 102: Divide the data to be transmitted into at least one data block.

In order to reduce the occupation of network bandwidth, the data to be transmitted needs to be divided into at least one data block. For example, the data to be transmitted is 1M, and the data to be transmitted may be divided into 10 data blocks.

Step 103: Determine the first data block that has not been transmitted and the second data block that has been transmitted in the at least one data block, and determine the first data index of the first data block and the index of the second data block Second data index.

In this embodiment, the data blocks that have been transferred to the data center may be stored in the physical machine. And determine which of the at least one data block has been transmitted and which has not been transmitted by using the transmitted data blocks stored locally on the physical machine.

Step 104: Send the first data block, the first data index, and the second data index to a data center, so that the data center stores the at least one data block.

In order to reduce the occupation of network bandwidth, only the first data block that has not been transmitted can be transmitted, and the second data block that has been transmitted can not be transmitted, only the second data index corresponding to the second data block is transmitted to the data The center only needs to be sufficient, thereby ensuring that the network occupancy is reduced by the network occupancy of the second data block compared to the prior art.

According to the solution of the above-mentioned embodiment, when the collected data to be transmitted is transmitted, by dividing the data to be transmitted into at least one data block, the first data block that has never been transmitted in the at least one data block and its The first data index and the second data index corresponding to the second data block that has already been transmitted can be transmitted to the data center without the need to transmit the second data block, thereby reducing the amount of data in the transmission process and reducing the network bandwidth. occupancy.

In an embodiment of the present invention, the division of the data to be transmitted can be realized in the following manner: further comprising: using the first formula to calculate the division length;

The first formula includes:

$\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; z</span>}\mathrm{<span\; class="notranslate">\; e</span>}$

Among them, k(s _i ) is used to represent the division length corresponding to s _i ; s _i is used to represent the i-th service provided by the platform; period(s _i ) is used to represent the sampling period configured for s _i ; status(s _i ) is used to characterize the status change granularity evaluation of s _i ; λ is used to represent the impact factor, which is a known constant; defaultult_size is used to represent the default length of the data block; where, when the type of the service is determined, status(s _i ) is a known parameter. Wherein, the setting of the defaultult_size is to ensure that the finally determined division length is at least the default length.

Wherein, the division length may be the number of bytes, for example, 10 bytes; or may be a specific space occupied, for example, 100KB.

Dividing the data to be transmitted into at least one data block includes: dividing the data to be transmitted into the at least one data block by using the division length.

For example, k(s _i ) is 10 bytes, and the data to be transmitted is 100 bytes, then the data to be transmitted can be divided into 10 data blocks, and each data block is 10 bytes.

For another example, k(s _i ) is 10 bytes, and the data to be transmitted is 99 bytes, then the data to be transmitted can be divided into 10 data blocks, and each data block in the first 9 data blocks The block is 10 bytes, and the last 1 data block is 9 bytes.

In an embodiment of the present invention, after the first data block, the first data index and the second data index are sent to the data center, in order to ensure the next data block transmission, the number of data blocks to be transmitted can be reduced Quantity may further include: after sending the first data block, the first data index and the second data index to the data center: sending the at least one data block and the at least one data block The data index corresponding to each data block in the database is stored locally.

In an embodiment of the present invention, the first data block that has not been transmitted and the second data block that has been transmitted in the at least one data block can be determined in the following manner, including:

Determine the first check code corresponding to each data block in each data block stored locally;

calculating a second check code corresponding to each data block in the at least one data block;

Traverse the first check code according to the second check code; use the data block corresponding to the second check code in the first check code as the second data block, and A data block corresponding to the second check code in the first check code is used as the first data block.

Wherein, the calculation method of the first check code corresponding to each data block in the locally stored data blocks is the same as the calculation method of the second check code corresponding to each data block in the at least one data block.

For example, the at least one data block includes 10 data blocks, which are respectively: data block 1, data block 2, data block 3, ..., data block 10, and the second check codes corresponding to each data block are respectively: A1 , A2, A3,..., A10. Assuming that the first check code stored locally includes the check codes of A1, A2, and A3, then it can be determined that data block 1, data block 2, and data block 3 are already transmitted data blocks, and data block 4, data block 5. Data block 6, ... data block 10 are data blocks that have never been transmitted.

In an embodiment of the present invention, the second check code corresponding to each data block in the at least one data block may be calculated in the following manner, and the technical method may include:

calculating a second check code corresponding to each data block in the at least one data block by using the second formula, the third formula and the fourth formula;

The second formula includes:

$\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; mod</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ;</span>$

The third formula includes:

$\mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; mod</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ;</span>$

The fourth formula includes:

Adler32(1,k)=A(1,k)+2 ¹⁶ B(1,k);

Among them, Adler32(1,k) is used to represent the second check code; A(1,k) is used to represent the first intermediate parameter, B(1,k) is used to represent the second intermediate parameter; data[j] is used to represent is used to represent the data corresponding to the jth byte in the current data block; M is used to represent a known constant; k is used to represent the number of bytes included in the current data block.

In the second check code calculated according to the above calculation method, if the second check code calculated according to the above calculation method is different from the first check code stored locally, it indicates that the different second check code The second data block corresponding to the verification code must have never been transmitted; if it is the same as the first verification code stored locally, it indicates that the second data block corresponding to the same second verification code may have been transmitted, and a unique The check value, for example, the MD5 value, is further checked. If the MD5 value is used to further check the result is the same, it indicates that the second data block corresponding to the second check code has been transmitted, otherwise, it has not been transmitted.

In an embodiment of the present invention, the second check code may also be directly calculated by means of a unique check value, so as to determine the first data block and the second data block.

Please refer to FIG. 2, the embodiment of the present invention also provides a data synchronization method applied to a data center, including:

Step 201: Obtain the first data block for the target type service sent by the current physical machine, the first data index corresponding to the first data block, and the second data index corresponding to the second data block; wherein, the first data The block is a data block that has not been transmitted by the current physical machine, and the second data block is a data block that has been transmitted by the current physical machine.

Step 202: Determine the second data according to each third data block included in the locally stored data copy, and the third data index corresponding to each third data block, and according to the second data index piece.

Data copies of each data block are stored locally in the data center, and data copies are a general technology to improve data access efficiency, system fault tolerance, and load balancing capabilities. The data copy includes not only the third data blocks, but also the third data index corresponding to each third data block, by searching the third data index for the same data index as the second data index, and using the same data The index determines the corresponding data block, and the determined data block is the second data block that has been transmitted.

Step 203: Store the first data block and the second data block.

In this embodiment, the threshold value of the amount of data corresponding to the service S _i can also be set, for example, the threshold value is 100, and when the number of data blocks included in the service reaches 100, the data block corresponding to the service is stored in the cloud in storage.

Furthermore, the data center stores the data requiring real-time performance in the real-time database and stores the data not requiring real-time performance in the ordinary database by judging the real-time requirement of the data.

In one embodiment of the present invention, due to the unreasonable number of data copies, it may cause storage pressure on the platform and waste a lot of storage space. Therefore, the number of data copies can be determined in the following manner, further including: The first data block and the second data block are copied as follows:

Among them, number is used to represent the target number of data copy replication for the first data block and the second data block; request_dead_lock is used to represent the multi-user request for the first data block and the second data block The number of service blocks caused by block data competition; all_request is used to represent the concurrent access amount corresponding to the target type of service; a is used to represent the scaling factor, which is a known constant; init_size is used to represent the initialization number of data copies.

The following takes the PAAS platform as an example for providing services. The PAAS platform includes at least one physical machine and a data center. Through the interaction between one of the physical machines and the data center, the data synchronization process is described in detail. Referring to Figure 3, the method may include the following steps:

Step 301: The physical machine configures corresponding sampling periods for different types of services.

Some new data may be generated at regular intervals on each physical machine, and these data need to be stored whenever new data is generated.

Assuming that there are 4 service types, configure the four service types as shown in Table 1 below:

Table 1:

Service type Sampling cycle/time (s) Type 1 60 Type 2 180 Type 3 300 Type 4 90

In this embodiment, different acquisition units can be used to collect data for the above four types of services respectively. Each acquisition unit, according to the sampling period configured in the above Table 1, whenever it reaches the corresponding sampling period, Execute collection of the service data of the corresponding type.

In this embodiment, the user also needs to register monitoring information, such as monitoring indicators, monitoring objects, monitoring methods, etc., with the Agent in the data center.

In order to provide better scalability, the Agent in the physical machine provides two expansion methods: the first is called by a script, the user transmits the monitoring script to the Agent, and the Agent calls the script to run the monitoring module and collect service status information. The second way is to call the Agent's API and extend it to achieve the purpose of obtaining service status information.

Step 302: When the sampling period corresponding to the target type service is reached, collect the data required to be transmitted by the target type service.

In this embodiment, the physical machine may include multiple virtual machines, and an agent may be configured on each virtual machine, and the agent may collect and process data.

Step 303: Calculate the division length according to the service type.

In this embodiment, the division length can be calculated by the following formula (1):

$\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; z</span>}\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Among them, k(s _i ) is used to represent the division length corresponding to s _i ; s _i is used to represent the i-th service provided by the platform; period(s _i ) is used to represent the sampling period configured for s _i ; status(s _i ) is used to represent the state change granularity evaluation of _si ; λ is used to represent the impact factor, which is a known constant; default_size is used to represent the default length of the data block.

Wherein, when the service type is determined, status(s _i ) is a known parameter.

Wherein, the setting of the defaultult_size is to ensure that the finally determined division length is at least the default length. For example, the minimum default length is 5 bytes.

Step 304: Using the calculated division length, divide the data to be transmitted into m data blocks.

Assume that the data to be transmitted by the target type service is divided into m data blocks as follows: in,

Step 305: Obtain each data block stored locally.

Each data block that has been sent to the data center and a data index corresponding to each data block are stored locally on the physical machine.

Step 306: Determine the first check code corresponding to each data block in the locally stored data blocks.

Step 307: Using the calculation method of the first check code, calculate the second check code corresponding to each of the m data blocks, and determine the m data blocks according to the first check code and the second check code The first data block that has not been transmitted and the second data block that has been transmitted.

In this embodiment, in order to ensure the consistency of the check code, it is necessary to calculate the second check code using the calculation method of the first check code.

Wherein, the calculation method may be to directly calculate a unique check value of each data block, and the unique check value may be an MD5 value. For example, the second check codes corresponding to the m data blocks are: A1, A2, A3, . . . , Am. The first verification code stored locally includes: A1, A2, and A3, then determine that data block 1, data block 2, and data block 3 are data blocks that have been transmitted, and data block 1, data block 2, and data block 3 As the second data block, use data block 4, data block 5, data block 6, ..., data block m as the first data block that has not been transmitted.

Since the calculation method of the unique check value is relatively complicated and takes a long time, the second check code can be calculated in the following way:

In an embodiment of the present invention, the second check code corresponding to each data block in the at least one data block may be calculated in the following manner, and the calculation method may include:

Utilize formula (2), formula (3) and formula (4) to calculate the second check code corresponding to each data block in m data blocks;

$\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; mod</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

$\mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; mod</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

Adler32(1,k)=A(1,k)+2 ¹⁶ B(1,k) (4)

Among them, Adler32(1,k) is used to represent the second check code; A(1,k) is used to represent the first intermediate parameter, B(1,k) is used to represent the second intermediate parameter; data[j] is used to represent is used to represent the data corresponding to the jth byte in the current data block; M is used to represent a known constant; k is used to represent the number of bytes included in the current data block.

In the second check code calculated according to the above calculation method, if the second check code calculated according to the above calculation method is different from the first check code stored locally, it indicates that the different second check code The second data block corresponding to the verification code must have never been transmitted; assuming that the second verification codes corresponding to the m data blocks are: A1, A2, A3, . . . , Am. A1, A2, and A3 are all located in the first check code stored locally. Then, A4, A5, A6, ..., Am have never been transmitted, and data block 4, data block 5, data block 6, ..., Data block m is determined as the second data block.

If it is the same as the first check code stored locally, it indicates that the second data block corresponding to the same second check code may have been transmitted, and a unique check value, such as an MD5 value, needs to be used for further check . Data block 1, data block 2, and data block 3 need to be further verified, and the MD5 values corresponding to data block 1, data block 2, and data block 3 are calculated, and the first check codes stored locally are calculated as A1 and A2 The MD5 values corresponding to the three data blocks of A3, if the MD5 values of the corresponding data blocks are the same, it indicates that the data block has been transmitted; otherwise, the data block has not been transmitted. For example, A1, A2, and A3 have all been transmitted, and data block 1, data block 2, and data block 3 are determined as the first data block.

Step 308: Determine the first data index corresponding to the first data block, and determine the second data index corresponding to the second data block.

Wherein, the first data index needs to be generated according to the first data block.

The second data index may be determined according to a locally stored index.

Step 309: Send the first data block, the first data index and the second data index to the data center, and store the first data block and the first data index locally.

In order to reduce the occupation of network bandwidth, only the first data block that has not been transmitted can be transmitted, and the second data block that has been transmitted can not be transmitted, only the second data index corresponding to the second data block is transmitted to the data The center only needs to be sufficient, thereby ensuring that the network occupancy is reduced by the network occupancy of the second data block compared to the prior art.

In this embodiment, after the first data block, the first data index and the second data index are sent to the data center, in order to ensure that the next data block transmission can reduce the number of data blocks to be transmitted, it may further include: Locally store at least one data block and a data index corresponding to each data block in the at least one data block. Wherein, the stored data may also include timestamps corresponding to the first data block and the second data block.

In this embodiment, the first data block, the first data index and the second data index may also be compressed, and the compressed data package is sent to the data center.

Step 310: The data center determines the second data block by using each third data block included in the stored data copy, the third data index corresponding to each third data block, and the second data index.

Since the second data block has been transmitted to the data center, the second data block and the second data index are stored in the data center.

Wherein, the second data block may be acquired through data copy.

Data copies of each data block are stored locally in the data center, and data copies are a general technology to improve data access efficiency, system fault tolerance, and load balancing capabilities. The data copy includes not only the third data blocks, but also the third data index corresponding to each third data block, by searching the third data index for the same data index as the second data index, and using the same data The index determines the corresponding data block, and the determined data block is the second data block that has been transmitted.

Step 311: According to the set threshold corresponding to the service of the target type, determine whether the number of the first data block and the number of the second data block is greater than the threshold, and if so, the number of the first data block and the second data block Stored in cloud storage, if not larger, then store the first data block and the second data block in the real-time database.

Further, a threshold value of the amount of data corresponding to the service S _i may also be set, for example, the threshold value is 100, and when the number of data blocks included in the service reaches 100, the data block corresponding to the service is stored in the cloud storage.

Step 312: Perform data copy replication on the first data block and the second data block.

The unreasonable number of data copies may cause storage pressure on the platform and waste a lot of storage space. Therefore, the number of data copies can be determined in the following way:

The number of data copies can be determined by the following formula (5):

Among them, number is used to represent the target number of data copy replication for the first data block and the second data block; request_dead_lock is used to represent the multi-user request for the first data block and the second data block The number of service blocks caused by block data competition; all_request is used to represent the concurrent access amount corresponding to the target type of service; a is used to represent the scaling factor, which is a known constant; init_size is used to represent the initialization number of data copies.

Please refer to FIG. 4, the embodiment of the present invention also provides a physical machine, which may include:

The configuration unit 401 is configured to respectively configure corresponding sampling periods for different types of services;

The collection unit 402 is configured to collect the data required to be transmitted by the target type service when the sampling period corresponding to the target type service is reached;

A dividing unit 403, configured to divide the data to be transmitted into at least one data block;

A determining unit 404, configured to determine a first data block that has not been transmitted in the at least one data block, a second data block that has been transmitted, and determine a first data index of the first data block, the second data block a second data index of the data block;

The sending unit 405 is configured to send the first data block, the first data index, and the second data index to a data center, so that the data center stores the at least one data block.

In an embodiment of the present invention, it may further include: a calculation unit, configured to calculate the division length by using the first formula;

The first formula includes:

$\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; z</span>}\mathrm{<span\; class="notranslate">\; e</span>}$

Among them, k(s _i ) is used to represent the division length corresponding to s _i ; s _i is used to represent the i-th service provided by the platform; period(s _i ) is used to represent the sampling period configured for s _i ; status(s _i ) is used to characterize the state change granularity evaluation of s _i ; λ is used to represent the impact factor, which is a known constant; default_size is used to represent the default length of the data block;

The division unit is specifically configured to use the division length to divide the data to be transmitted into the at least one data block;

In an embodiment of the present invention, it may further include: a storage unit configured to locally store the at least one data block and a data index corresponding to each data block in the at least one data block;

In one embodiment of the present invention, the determining unit includes:

A determining module, configured to determine a first check code corresponding to each data block in each data block stored locally;

A calculation module, configured to calculate a second check code corresponding to each data block in the at least one data block;

A traversal module, configured to traverse the first check code according to the second check code; use the data block corresponding to the second check code in the first check code as the second check code For a data block, a data block corresponding to the second check code that is not located in the first check code is used as the first data block.

Please refer to FIG. 5, the embodiment of the present invention also provides a data center, which may include:

The obtaining unit 501 is configured to obtain the first data block sent by the current physical machine for the service of the target type, the first data index corresponding to the first data block, and the second data index corresponding to the second data block; wherein, the The first data block is a data block that has not been transmitted by the current physical machine, and the second data block is a data block that has been transmitted by the current physical machine;

The determining unit 502 is configured to determine the said second data block;

The storage unit 503 is configured to store the first data block and the second data block.

In an embodiment of the present invention, it may further include: a copying unit, configured to perform the following target number of data copies on the stored first data block and the second data block:

Among them, number is used to represent the target number of data copy replication for the first data block and the second data block; request_dead_lock is used to represent the multi-user request for the first data block and the second data block The number of service blocks caused by block data competition; all_request is used to represent the concurrent access amount corresponding to the target type of service; a is used to represent the scaling factor, which is a known constant; init_size is used to represent the initialization number of data copies.

Please refer to FIG. 6 , an embodiment of the present invention also provides a data synchronization system, which may include: any one of the data centers 50 described above, and at least one physical machine 40 described above.

In summary, each embodiment of the present invention can at least achieve the following beneficial effects:

1. In the embodiment of the present invention, when the collected data to be transmitted is transmitted, by dividing the data to be transmitted into at least one data block, the first data that has not been transmitted in the at least one data block block and its first data index, and the second data index corresponding to the second data block that has already been transmitted can be transmitted to the data center without the need to transmit the second data block, thereby reducing the amount of data in the transmission process, which in turn can reduce Occupancy of network bandwidth.

2. In the embodiment of the present invention, the Agent in the physical machine can provide flexible expansion functions, and the user can dynamically expand the data monitoring and collection of the Agent through interface expansion or script language calling. Users do not need to make changes to the entire platform system because of the introduction of customized services.

3. In the embodiment of the present invention, the service-oriented multi-copy data storage strategy, on the basis of ensuring real-time storage and access of data, not only reduces the redundant number of data copies, but also improves the concurrent access of the system.

The information exchange and execution process among the units in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific content can refer to the description in the method embodiment of the present invention, and will not be repeated here.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or sequence. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a" does not exclude the presence of additional same elements in the process, method, article or apparatus comprising said element.

Those of ordinary skill in the art can understand that all or part of the steps to realize the above method embodiments can be completed by program instructions related hardware, and the aforementioned programs can be stored in a computer-readable storage medium. When the program is executed, the It includes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other various media that can store program codes.

Finally, it should be noted that: the above descriptions are only preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, and are not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (10)

1. A data synchronization method, characterized in that, being applied to a physical machine, corresponding sampling periods are respectively configured for different types of services; the method comprises: When the sampling period corresponding to the target type service is reached, collect the data required to be transmitted by the target type service; dividing the data to be transmitted into at least one data block; Determining the first data block that has not been transmitted and the second data block that has been transmitted in the at least one data block, and determining the first data index of the first data block and the second data of the second data block index; sending the first data block, the first data index, and the second data index to a data center, so that the data center stores the at least one data block. 2. The method of claim 1, wherein, Before dividing the data to be transmitted into at least one data block, it further includes: using a first formula to calculate the division length; The first formula includes: $\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; z</span>}\mathrm{<span\; class="notranslate">\; e</span>}$ Among them, k(s _i ) is used to represent the division length corresponding to s _i ; s _i is used to represent the i-th service provided by the platform; period(s _i ) is used to represent the sampling period configured for s _i ; status(s _i ) is used to characterize the state change granularity evaluation of s _i ; λ is used to represent the impact factor, which is a known constant; default_size is used to represent the default length of the data block; Dividing the data to be transmitted into at least one data block includes: dividing the data to be transmitted into the at least one data block by using the division length; and / or, After sending the first data block, the first data index, and the second data index to the data center, further comprising: sending the at least one data block, and each of the at least one data block The data index corresponding to the data block is stored locally; and / or, The determining the first data block that has not been transmitted and the second data block that has been transmitted in the at least one data block includes: Determine the first check code corresponding to each data block in each data block stored locally; calculating a second check code corresponding to each data block in the at least one data block; Traverse the first check code according to the second check code; use the data block corresponding to the second check code in the first check code as the second data block, and A data block corresponding to the second check code in the first check code is used as the first data block. 3. The method according to claim 2, wherein the calculating the second check code corresponding to each data block in the at least one data block comprises: calculating a second check code corresponding to each data block in the at least one data block by using the second formula, the third formula and the fourth formula; The second formula includes: $\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; mod</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ;</span>$ The third formula includes: $\mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; mod</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; ;</span>$ The fourth formula includes: Adler32(1,k)=A(1,k)+2 ¹⁶ B(1,k); Among them, Adler32(1,k) is used to represent the second check code; A(1,k) is used to represent the first intermediate parameter, B(1,k) is used to represent the second intermediate parameter; data[j] is used to represent is used to represent the data corresponding to the jth byte in the current data block; M is used to represent a known constant; k is used to represent the number of bytes included in the current data block. 4. A data synchronization method, characterized in that it is applied to a data center, comprising: Obtain the first data block for the target type service sent by the current physical machine, the first data index corresponding to the first data block, and the second data index corresponding to the second data block; wherein, the first data block is the A data block that has never been transmitted by the current physical machine, and the second data block is a data block that has been transmitted by the current physical machine; determining the second data block according to each third data block included in the locally stored data copy, and a third data index corresponding to each of the third data blocks, and according to the second data index; The first data block and the second data block are stored. 5. The method according to claim 4, further comprising: performing the following target number of data copies on the stored first data block and the second data block: Among them, number is used to represent the target number of data copy replication for the first data block and the second data block; request_dead_lock is used to represent the multi-user request for the first data block and the second data block The number of service blocks caused by block data competition; all_request is used to represent the concurrent access amount corresponding to the target type of service; a is used to represent the scaling factor, which is a known constant; init_size is used to represent the initialization number of data copies. 6. A physical machine, characterized in that, comprising: The configuration unit is configured to respectively configure corresponding sampling periods for different types of services; A collection unit, configured to collect the data required to be transmitted by the target type service when the sampling period corresponding to the target type service is reached; a division unit, configured to divide the data to be transmitted into at least one data block; A determining unit, configured to determine a first data block that has not been transmitted in the at least one data block, a second data block that has been transmitted, and determine the first data index of the first data block, the second data block the second data index of the block; A sending unit, configured to send the first data block, the first data index, and the second data index to a data center, so that the data center stores the at least one data block. 7. The physical machine according to claim 6, wherein: Further comprising: a calculation unit, configured to calculate the division length by using the first formula; The first formula includes: $\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; p</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; *</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; z</span>}\mathrm{<span\; class="notranslate">\; e</span>}$ Among them, k(s _i ) is used to represent the division length corresponding to s _i ; s _i is used to represent the i-th service provided by the platform; period(s _i ) is used to represent the sampling period configured for s _i ; status(s _i ) is used to characterize the state change granularity evaluation of s _i ; λ is used to represent the impact factor, which is a known constant; default_size is used to represent the default length of the data block; The division unit is specifically configured to divide the data to be transmitted into the at least one data block by using the division length; and / or, Further comprising: a storage unit, configured to locally store the at least one data block and a data index corresponding to each data block in the at least one data block; and / or, The determination unit includes: A determining module, configured to determine a first check code corresponding to each data block in each data block stored locally; A calculation module, configured to calculate a second check code corresponding to each data block in the at least one data block; A traversal module, configured to traverse the first check code according to the second check code; use the data block corresponding to the second check code in the first check code as the second check code For a data block, a data block corresponding to the second check code that is not located in the first check code is used as the first data block. 8. A data center, characterized in that, comprising: The acquiring unit is configured to acquire the first data block sent by the current physical machine for the service of the target type, the first data index corresponding to the first data block, and the second data index corresponding to the second data block; wherein, the first A data block is a data block that has not been transmitted by the current physical machine, and the second data block is a data block that has been transmitted by the current physical machine; A determining unit, configured to determine the first data block according to each third data block included in the locally stored data copy, and the third data index corresponding to each third data block, and according to the second data index Two data blocks; A storage unit, configured to store the first data block and the second data block. 9. The data center according to claim 8, further comprising: a replication unit configured to perform the following target number of data copies on the stored first data block and the second data block: Among them, number is used to represent the target number of data copy replication for the first data block and the second data block; request_dead_lock is used to represent the multi-user request for the first data block and the second data block The number of service blocks caused by block data competition; all_request is used to represent the concurrent access amount corresponding to the target type of service; a is used to represent the scaling factor, which is a known constant; init_size is used to represent the initialization number of data copies. 10. A data synchronization system, characterized by comprising: the data center according to claim 8 or 9 above, and at least one physical machine according to claim 6 or 7 above.