CN113010602A

CN113010602A - Data synchronization method, device and system

Info

Publication number: CN113010602A
Application number: CN202110270274.2A
Authority: CN
Inventors: 蒋源德; 年国栋; 龚少成
Original assignee: Beijing Sankuai Online Technology Co Ltd
Current assignee: Beijing Sankuai Online Technology Co Ltd
Priority date: 2021-03-12
Filing date: 2021-03-12
Publication date: 2021-06-22

Abstract

The present specification discloses a data synchronization method, apparatus and system, which can determine a data set to be synchronized first and create transmission instances corresponding to each receiving database. And then, determining the data packet to be transmitted corresponding to the transmission instance according to a preset data receiving rule. And finally, executing a transmission example corresponding to the receiving database, uploading the data packet to be transmitted corresponding to the transmission example and the identification of the receiving database to a sending server, so that the sending server sends the data packet to be transmitted and the identification of the receiving database to the receiving server through a safety special line, and the receiving server writes the data packet to be transmitted into the receiving database. By determining the data packets to be transmitted corresponding to the receiving databases and writing the data packets to be transmitted into the corresponding receiving databases respectively, the receiving databases do not need to poll to acquire data, the receiving databases are prevented from acquiring data which is not required by the receiving databases, and the efficiency of data synchronization is improved.

Description

Data synchronization method, device and system

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data synchronization method, apparatus, and system.

Background

At present, in order to ensure security in data transmission, there are various ways. One of them is to connect the devices needing data transmission from the network structure through a dedicated line to avoid external attacks.

In the prior art, for a scenario that data synchronization between databases is required, a method for data transmission by a data sending party and a data receiving party through respective File Transfer Protocol (FTP) servers is generally adopted to ensure security. The FTP servers are connected by adopting a special line, and the data sending party, the data receiving party and the respective FTP servers are arranged in respective internal networks, so that the respective safety is improved. And even only the data transmitted by the FTP server can be accepted so as to further ensure the security.

Fig. 1 is a schematic diagram of a conventional network structure, where the left side is a data sender and its FTP server, the right side is a data receiver and its FTP server, the two FTP servers are connected by a dedicated line, and the devices on the left and right sides are respectively located in their own intranet.

However, in the conventional data synchronization method, in order to ensure the security of data transmission, it is necessary to transmit data between FTP servers connected to dedicated lines, and a data receiving side needs to poll the corresponding FTP server to acquire synchronization data, which is inefficient. Moreover, because the FTP server generally serves a plurality of devices, there are situations in which data that is not needed by the FTP server is obtained, which results in inefficient synchronization of existing data.

Disclosure of Invention

Embodiments of the present specification provide a data synchronization method, apparatus, and system, so as to partially solve the problems in the prior art.

The embodiment of the specification adopts the following technical scheme:

the data synchronization method provided by the present specification includes:

the local database determines a data set to be synchronized;

aiming at each receiving database, creating a transmission instance corresponding to the receiving database, wherein the transmission instance carries an identifier of the receiving database;

determining a data packet to be transmitted corresponding to the transmission example from the data set to be synchronized according to a preset data receiving rule of the receiving database;

executing a transmission example corresponding to the receiving database, uploading a data packet to be transmitted corresponding to the transmission example and the identification of the receiving database to a sending server, so that the sending server sends the data packet to be transmitted and the identification of the receiving database to a receiving server through a safety special line, and the receiving server writes the data packet to be transmitted into the receiving database according to the identification of the receiving database.

Optionally, for each receiving database, creating a transmission instance corresponding to the receiving database, specifically including:

determining a transmission task according to the data set to be synchronized;

and aiming at each receiving database, creating a transmission instance corresponding to the receiving database contained in the transmission task, wherein the transmission instance belongs to the transmission task.

Optionally, determining, according to a preset data receiving rule of the receiving database, a to-be-transmitted data packet corresponding to the transmission instance from the to-be-synchronized data set, specifically including:

determining the data receiving rules of the receiving database according to the corresponding relation between the stored receiving databases and the data receiving rules;

judging whether data matched with a data receiving rule of the receiving database exists in the data set to be synchronized;

if so, taking the matched data as the data packet to be transmitted corresponding to the transmission example;

if not, determining that the transmission instance of the receiving database is executed.

Optionally, executing a transmission instance corresponding to the receiving database, and uploading a to-be-transmitted data packet corresponding to the transmission instance and an identifier of the receiving database to a sending server, specifically including:

when the data package to be transmitted corresponding to each transmission example in the transmission task is determined, grouping the transmission examples according to the number of preset parallel examples;

and sequentially executing the transmission examples of each group through idle threads, uploading the data packets to be transmitted corresponding to the transmission examples of each group and the identification of the corresponding receiving database to a sending server respectively, and determining that the transmission task is completely executed after the transmission examples of each group are completely executed.

Optionally, uploading to the sending server specifically includes:

determining a storage path of each receiving database preset in the sending server;

and sending the data package to be transmitted corresponding to the transmission example and the identification of the receiving database to the storage path for storage, so that the sending server sends the data package to be transmitted corresponding to the transmission example and the identification of the receiving database to the receiving server when monitoring that the data in the storage path changes.

The data synchronization method provided by the present specification includes:

the receiving server is used for receiving the data of each transmission example sent by the sending server through a safe private line, wherein the data of the transmission example at least comprises a data package to be transmitted corresponding to the transmission example and an identifier of a receiving database corresponding to the transmission example;

for a data packet to be transmitted corresponding to each transmission example, writing the data packet to be transmitted into a corresponding receiving database according to the identification of the receiving database corresponding to the transmission example;

wherein the data of the transmission instance is uploaded to the sending server from a local database by:

the local database creates a transmission example corresponding to each receiving database, determines a data packet to be transmitted corresponding to each transmission example from the data set to be synchronized based on the data receiving rule of each receiving database, and uploads the data of each transmission example to the sending server by executing each transmission example.

Optionally, writing the data packet to be transmitted into the corresponding receiving database according to the identifier of the receiving database corresponding to the transmission instance, specifically including:

determining the attribute of the receiving database according to the identifier of the receiving database corresponding to the transmission instance, wherein the attribute at least comprises: one of a data structure or a database structure;

carrying out format conversion on the data packet to be transmitted according to the attribute of the receiving database;

and storing the converted data packet into the receiving database, so that the receiving database performs service processing according to the converted data.

Optionally, the method further comprises:

judging whether the data packet to be transmitted is data required by service processing or not according to the data file name in the data packet to be transmitted;

if yes, writing the data packet to be transmitted into a corresponding receiving database;

and if not, not storing the data packet to be transmitted.

A data synchronization system provided by the present specification, the system comprising: local database, sending server and receiving server, wherein:

the local database determines a data set to be synchronized; aiming at each receiving database, creating a transmission instance corresponding to the receiving database, wherein the transmission instance carries an identifier of the receiving database; determining a data packet to be transmitted corresponding to the transmission example from the data set to be synchronized according to a preset data receiving rule of the receiving database; executing a transmission example corresponding to the receiving database, and uploading a data packet to be transmitted corresponding to the transmission example and the identification of the receiving database to a sending server;

the sending server sends the data packet to be transmitted corresponding to each transmission instance and the identification of the receiving database corresponding to the data packet to be transmitted to the receiving server through a safe special line;

and the receiving server writes the data packet to be transmitted into the corresponding receiving database according to the identification of the receiving database corresponding to the transmission example aiming at the data packet to be transmitted corresponding to each transmission example.

The present specification provides a data synchronization apparatus, including:

the first determining module is used for determining a data set to be synchronized through a local database;

the creating module is used for creating a transmission instance corresponding to each receiving database, wherein the transmission instance carries the identification of the receiving database;

the second determining module is used for determining the data package to be transmitted corresponding to the transmission example from the data set to be synchronized according to a preset data receiving rule of the receiving database;

and the execution and writing module is used for executing a transmission example corresponding to the receiving database, uploading a data packet to be transmitted corresponding to the transmission example and the identification of the receiving database to a sending server, so that the sending server sends the data packet to be transmitted and the identification of the receiving database to the receiving server through a safety special line, and the receiving server writes the data packet to be transmitted into the receiving database according to the identification of the receiving database.

The present specification provides a data synchronization apparatus, including:

the receiving module is used for receiving data of each transmission example sent by a sending server through a safe special line through a receiving server, wherein the data of the transmission example at least comprises a data package to be transmitted corresponding to the transmission example and an identification of a receiving database corresponding to the transmission example;

the writing module writes the data packet to be transmitted into the corresponding receiving database according to the identification of the receiving database corresponding to the transmission example aiming at the data packet to be transmitted corresponding to each transmission example, wherein the data of the transmission example is uploaded to the sending server from the local database in the following mode: the local database creates a transmission example corresponding to each receiving database, determines a data packet to be transmitted corresponding to each transmission example from the data set to be synchronized based on the data receiving rule of each receiving database, and uploads the data of each transmission example to the sending server by executing each transmission example.

The present specification provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the data synchronization method described above.

The present specification provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the data synchronization method described above.

The embodiment of the specification adopts at least one technical scheme which can achieve the following beneficial effects:

in this specification, the local database may first determine a data set to be synchronized, and for each receiving database, create a transmission instance corresponding to the receiving database. And then, determining the data package to be transmitted corresponding to the transmission instance from the data set to be synchronized according to a preset data receiving rule of the receiving database. And finally, executing a transmission example corresponding to the receiving database, uploading the data packet to be transmitted corresponding to the transmission example and the identification of the receiving database to a sending server, so that the sending server sends the data packet to be transmitted and the identification of the receiving database to the receiving server through a safety special line, and the receiving server writes the data packet to be transmitted into the receiving database according to the identification of the receiving database. The data packets to be transmitted corresponding to the receiving databases are determined, and the data packets to be transmitted are respectively written into the corresponding receiving databases, so that the receiving databases do not need to poll to acquire data, and the timeliness of data transmission is high. Meanwhile, each receiving database is prevented from acquiring data which is not required by the receiving database, and the efficiency of data synchronization is improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the specification and are incorporated in and constitute a part of this specification, illustrate embodiments of the specification and together with the description serve to explain the specification and not to limit the specification in a non-limiting sense. In the drawings:

fig. 1 is a schematic diagram of a conventional network structure for data transmission;

fig. 2 is a schematic diagram of a data synchronization process provided in an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of another data synchronization process provided by embodiments of the present disclosure;

fig. 4 is a schematic architecture diagram of a data synchronization system according to an embodiment of the present disclosure;

fig. 5 is a schematic diagram of interaction between devices in a data synchronization system according to an embodiment of the present disclosure;

FIG. 6 is a block diagram of another data synchronization system provided in an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of a data synchronization apparatus provided in an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of a data synchronization apparatus provided in an embodiment of the present disclosure;

fig. 9 is a schematic structural diagram of an electronic device implementing a data synchronization method according to an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below with reference to the specific embodiments of the present disclosure and the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present disclosure, and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present specification without any creative effort belong to the protection scope of the present specification.

The technical solutions provided by the embodiments of the present description are described in detail below with reference to the accompanying drawings.

Fig. 2 is a schematic diagram of a data synchronization process provided in an embodiment of the present specification, including:

s100: the local database determines the data set to be synchronized.

The data synchronization method provided by the present specification realizes data synchronization of databases of both parties by transmitting data in a local database of a data transmitting party to a receiving database of a data receiving party. Therefore, when data synchronization is performed in this specification, a data set to be synchronized in a local database of a data sender may be determined first, so as to perform data synchronization through subsequent steps.

Specifically, the data sender may transmit data to the data receiver at a fixed time, and when the set time point is reached, the data sender may determine the data to be transmitted from the local database, and use the data to be transmitted as the data set to be synchronized. Wherein the data set to be synchronized may be data stored at any time before the set point in time.

Or in another embodiment of the present specification, the data sender may also determine, in response to a user operation, data selected by the user in the local database as the data set to be synchronized.

It should be noted that the data synchronization method may be specifically executed by a synchronization task server corresponding to a local database of a data sending party, where the synchronization task server is used to execute a data synchronization task, and may be a single server or a system composed of multiple servers, such as a distributed server system, and may be a physical server device or a cloud server, and of course, the data sending party may also include other servers, which is not limited in this specification. For convenience of description, a local database is used as an execution subject in the description.

Further, when the data set to be synchronized is automatically determined in a timing manner, a Software Development Kit (SDK) may be pre-installed in the synchronization task server, and the SDK includes a code for executing a timing task. In the data synchronization process, the data set to be synchronized can be determined from the local database at regular time by running the code in the SDK.

When the data set to be synchronized is determined manually based on a user operation mode, specifically, a HyperText Transfer Protocol (HTTP) interface is further provided in the synchronization task server, and a user can select data to be transmitted on a terminal interface and transmit the data to the synchronization task server through the HTTP interface, so that the synchronization task server responds to the selected operation of the user to determine the data set to be synchronized.

Of course, in this specification, if there are a plurality of local databases in the data sending side, and each local database corresponds to a different synchronization task server, the data synchronization method in this specification may be executed for each local database. If the data sending party has a plurality of local databases, and the plurality of local databases all correspond to the same synchronization task server, the synchronization task server can determine the data to be transmitted from the plurality of local databases to serve as the data set to be synchronized.

S102: and aiming at each receiving database, creating a transmission instance corresponding to the receiving database, wherein the transmission instance carries the identification of the receiving database.

S104: and determining the data packet to be transmitted corresponding to the transmission example from the data set to be synchronized according to a preset data receiving rule of the receiving database.

In order to enable the data to be transmitted to the data receiving party, each receiving database can only obtain the data required by the receiving database, and other redundant data are prevented from being downloaded. In one or more embodiments of the present disclosure, the data sender may determine, from the local database, the data packets to be distributed to each receiving database for transmission to each receiving database of the data receiver, respectively.

Specifically, after the data set to be synchronized is determined, the local database may determine a transmission task according to the data set to be synchronized, so as to transmit the data set through the transmission task.

Then, for each receiving database of the data receiving party, the local database may create a transmission instance corresponding to the receiving database in the transmission task, where the transmission instance carries an identifier of the receiving database, and may further include table name information to be written into a data table in the receiving database, and the like. Also, the transmission task may include several transmission instances corresponding to each receiving database.

Finally, the local database can determine the data receiving rule corresponding to the receiving database according to the corresponding relationship between the pre-stored receiving database and the data receiving rule. And judging whether the data set to be synchronized has data matched with the data receiving rule of the receiving database, if so, taking the matched data as the data packet to be transmitted corresponding to the transmission instance, otherwise, determining that the transmission instance of the receiving database is completely executed and no data needs to be transmitted to the receiving database.

S106: executing a transmission example corresponding to the receiving database, uploading a data packet to be transmitted corresponding to the transmission example and the identification of the receiving database to a sending server, so that the sending server sends the data packet to be transmitted and the identification of the receiving database to a receiving server through a safety special line, and the receiving server writes the data packet to be transmitted into the receiving database according to the identification of the receiving database.

In this specification, after determining the data packet to be transmitted corresponding to the receiving database, the data packet to be transmitted may be transmitted to the data receiving side, so that the data receiving side writes the data packet into the receiving database, and data synchronization is completed.

Specifically, the local database may execute a transmission instance corresponding to the receiving database, and upload the to-be-transmitted data packet corresponding to the transmission instance and the identifier of the receiving database to a sending server of a data sending party, so that the sending server sends the to-be-transmitted data packet and the identifier of the receiving database to a receiving server of a data receiving party through a secure dedicated line, and the receiving server writes the to-be-transmitted data packet into the receiving database according to the identifier of the receiving database. The sending server may be an FTP server of a data sending party, and the receiving server may be an FTP server of a data receiving party.

Further, when the data packet to be transmitted corresponding to each transmission instance in the transmission task is determined, the local database can group the transmission instances according to the preset number of the parallel instances, and determine a plurality of transmission instances contained in each group. And then, sequentially executing the transmission examples of each group through idle threads, uploading the data packages to be transmitted corresponding to the transmission examples of each group and the identification of the corresponding receiving database to a sending server respectively, and determining that the transmission task is completely executed after the transmission examples of each group are completely executed. The preset number of parallel instances is usually set according to the number of idle threads in history, and this specification does not limit this. The specific implementation process can be referred to in the prior art, and is not described in detail herein.

Furthermore, for each packet, before the transmission instances in the packet are sequentially executed by the idle threads, the local database may further detect the number of current idle threads, and if the number of current idle threads is not less than the preset number of parallel instances, the transmission instances in the packet may be executed in parallel by the current idle threads. If the number of the current idle threads is smaller than the preset number of the parallel instances, the corresponding number of the transmission instances can be selected from the group, the current idle threads are executed in parallel, and when a new idle thread appears, the rest transmission instances in the group are executed continuously. For example, assuming that each packet includes 10 transmission instances, when executing each transmission instance included in the first packet, if there are currently 7 idle threads, the current idle thread executes the 7 transmission instances in the packet first, and when there is an idle thread, the remaining 3 transmission instances in the packet are executed again. After each transmission instance contained in the packet is executed, the transmission instances in the next packet are executed in sequence.

In addition, when the data package to be transmitted corresponding to the transmission instance and the identifier of the receiving database are uploaded to the sending server in the present specification, the local database may first determine, for each receiving database, a storage path of the receiving database preset in the sending server. And then, the data package to be transmitted corresponding to the transmission example and the identification of the receiving database are sent to the determined storage path for storage, so that the sending server sends the data package to be transmitted corresponding to the transmission example and the identification of the receiving database to a receiving server of a data receiving party when monitoring the data change in the storage path.

When the transmission instance is executed, the local database can generate a corresponding execution code according to the transmission instance. And executing a transmission example through the execution code, downloading a data package to be transmitted corresponding to the transmission example from the local database, and uploading the data package to be transmitted and the identification of the receiving database to a corresponding storage path of a sending server. Of course, the local database may also upload, to the sending server, table name information of a data table in the receiving database, which needs to be written in by the packet to be transmitted, and the like.

Based on the data synchronization method shown in fig. 2, the local database may determine the data set to be synchronized first, and create a transmission instance corresponding to each receiving database. And then, determining the data package to be transmitted corresponding to the transmission instance from the data set to be synchronized according to a preset data receiving rule of the receiving database. And finally, executing a transmission example corresponding to the receiving database, uploading the data packet to be transmitted corresponding to the transmission example and the identification of the receiving database to a sending server, so that the sending server sends the data packet to be transmitted and the identification of the receiving database to the receiving server through a safety special line, and the receiving server writes the data packet to be transmitted into the receiving database according to the identification of the receiving database. The data packets to be transmitted corresponding to the receiving databases are determined, and the data packets to be transmitted are respectively written into the corresponding receiving databases, so that the receiving databases do not need to poll to acquire data, and the timeliness of data transmission is high. Meanwhile, each receiving database is prevented from acquiring data which is not required by the receiving database, and the efficiency of data synchronization is improved.

The data synchronization method provided by the specification can be applied to a process of transmitting large-batch data with high security, such as a data transmission process between financial platforms, or can also be applied to a process of transmitting confidential documents, and the data synchronization method can be applied to a process of transmitting the confidential documents through a special line.

For the data synchronization method shown in fig. 2, the present specification also correspondingly provides an execution flow on the receiving server side, as shown in fig. 3.

Fig. 3 is another schematic diagram of a data synchronization process provided in an embodiment of the present disclosure, where a receiving server is used as an execution main body, and the data synchronization process may specifically include the following steps:

s200: and the receiving server is used for receiving the data packet to be transmitted corresponding to each transmission example and the identification of the receiving database corresponding to each transmission example, which are transmitted by the sending server through the special safety line.

The data synchronization method provided in this specification may also be executed by a receiving server as an execution subject, where the receiving server may be a single server, or may be a cluster formed by multiple servers, such as a distributed server system.

In this specification, the receiving server is generally an FTP server on the data receiving side, and the servers corresponding to the receiving databases on the data receiving side are in the same intranet. In order to ensure the safety of data transmission, data transmission can be carried out between a receiving server of a data receiving party and a sending server of a data sending party through a safety special line.

Specifically, the receiving server may sequentially receive data of each transmission instance, where the data of each transmission instance includes a to-be-transmitted data package corresponding to the transmission instance and an identifier of a receiving database corresponding to the transmission instance.

Further, the data of each transmission instance can be uploaded to the sending server of the data sender from the local database of the data sender by the following method:

the local database of the data sender can create a transmission instance corresponding to each receiving database, determine a to-be-transmitted data packet corresponding to each transmission instance from the to-be-synchronized data set based on the data receiving rule of each receiving database, and respectively upload the data of each transmission instance to the sending server of the data sender by executing each transmission instance. The sending server of the data sending party is usually an FTP server of the data sending party, and data transmission is performed between the sending server of the data sending party and the FTP server of the data receiving party through a special safety line.

In this specification, the transmission process of specifically transmitting the data of each transmission instance to the sending server has been described in detail in the above step S100 to step S106, and this specification is not described herein again, and reference may be made to the above specifically.

S202: and aiming at the data packet to be transmitted corresponding to each transmission example, writing the data packet to be transmitted into the corresponding receiving database according to the identification of the receiving database corresponding to the transmission example.

In one or more embodiments of the present description, after the receiving server receives the data of each transmission instance, the to-be-transmitted data package corresponding to each transmission instance may be written into each receiving database, respectively.

Specifically, for the received data packet to be transmitted corresponding to each transmission instance, the receiving server may write the data packet to be transmitted into the corresponding receiving database according to the identifier of the receiving database corresponding to the transmission instance.

Further, since the data structure of the data packet to be transmitted sent by the data sender is not necessarily the same as the data structure of the data stored in the receiving database of the data receiver, if the data packet to be transmitted is the data stored in the relational database and the receiving database is a Hadoop Distributed File System (Hadoop) Distributed database, the receiving database cannot directly store the data packet to be transmitted.

In this specification, the receiving server may further determine, according to the identifier of the receiving database corresponding to the transmission instance, an attribute of the receiving database from a correspondence relationship between the identifier of each receiving database and the attribute of each receiving database, where the attribute includes at least one of a data structure of data stored in the receiving database or a database structure of the receiving database. And then, the receiving server can perform format conversion on the data packet to be transmitted according to the attribute of the receiving database, and store the converted data packet into the receiving database, so that the receiving database performs service processing according to the converted data.

When format conversion is performed on the data packet to be transmitted according to the attribute of the receiving database, the receiving server can also judge whether the data structure of the data packet to be transmitted conforms to the attribute of the receiving database according to the data structure of the data packet to be transmitted and the attribute of the receiving database, namely, whether the data packet to be transmitted can be directly stored in the receiving database, and when the data structure of the data packet to be transmitted is determined not to conform to the attribute of the receiving database, format conversion is performed on the data packet to be transmitted according to the data structure of the data packet to be transmitted and the attribute of the receiving database. When the data structure of the data packet to be transmitted is determined to be in accordance with the attribute of the receiving database, the data packet to be transmitted can be directly stored in the receiving database.

Furthermore, in this specification, the receiving server may further include a monitoring module, and the monitoring module monitors information change in the receiving server, so that when it is monitored that the receiving server receives data of each transmission instance, the attribute of each receiving database may be determined according to the received data of each transmission instance. The attribute may further include table name information of a data table in a receiving database, where the table name information may be determined according to a data file name of a data package to be transmitted corresponding to the transmission instance.

Based on the data synchronization method shown in fig. 3, the receiving server may first receive data of each transmission instance sent by the sending server through the dedicated security line, and write the data packet to be transmitted into the corresponding receiving database according to the identifier of the receiving database corresponding to each transmission instance for the data packet to be transmitted corresponding to each transmission instance. And the data of each transmission instance is determined from the data set to be synchronized by the local database and is uploaded to the sending server respectively. The data packets to be transmitted corresponding to the transmission examples are respectively written into the receiving databases corresponding to the transmission examples, the receiving databases do not need to poll to acquire data, and the timeliness of data transmission is high. Meanwhile, each receiving database is prevented from acquiring data which is not required by the receiving database, and the efficiency of data synchronization is improved.

In addition, in this specification, the receiving server may be a distributed server system, and includes a first server configured to receive data of each transmission instance, and a second server configured to monitor that information change occurs in the first server, and when it is monitored by the second server that information change occurs in the first server, that is, when data of each transmission instance sent by the sending server is received, the data of each transmission instance may be obtained by the second server, and the data may be written into a corresponding receiving database according to an identifier of a receiving database corresponding to each transmission instance.

In step S202 in this specification, the data of the transmission instance received by the receiving server may also be a test file, for example, used for testing whether a transmission failure occurs in the safety dedicated line, or testing the transmission duration required for data transmission. For such test files, the test files do not need to be written into each receiving database, so that the receiving server can also judge whether the data package to be transmitted is the data required by the receiving database for service processing according to the name of the data file in the data package to be transmitted. And if the data packet to be transmitted is not required for the service processing of the receiving database, the data packet to be transmitted is not stored and is not required to be written into the corresponding receiving database.

In an embodiment of the present specification, after the data of the transmission instance transmitted through the secure private line is monitored by the monitoring module or the second server, the occurred information change may be converted into an event stream, and the attribute of the event stream is determined according to the data of the transmission instance. The attribute of the event stream may include an attribute of a receiving database corresponding to the transmission instance, table name information of a data table of the written receiving database, and the like. And then, writing the data packet to be transmitted of the transmission instance corresponding to the event stream into a corresponding receiving database through a data synchronization tool.

Or in another embodiment of this specification, the receiving server may also create a write task, create, for each received transmission instance, a write instance corresponding to the transmission instance in the write task, and then sequentially process each write instance in the write task through an idle thread, and write the to-be-transmitted data packet corresponding to each transmission instance in the corresponding receiving database.

Based on the data synchronization methods shown in fig. 2 and fig. 3, the present specification also correspondingly provides a data synchronization system, as shown in fig. 4.

Fig. 4 is a schematic structural diagram of a data synchronization system provided in an embodiment of this specification, where the system includes a local database 300 of a data sender, a sending server 302, and a receiving server 304 of a data receiver, and in the system, the system further includes a synchronization task server corresponding to the local database 300 of the data sender, a plurality of receiving databases 306 of the data receiver, and a server corresponding to each receiving database 306. The sending server 302 of the data sending party, the local database 300 and the synchronous task server corresponding to the local database 300 are in the same intranet, the receiving server 304 of the data receiving party, the receiving database 306 and the server corresponding to the receiving database 306 are in the same intranet, and data transmission is performed between the sending server 302 of the data sending party and the receiving server 304 of the data receiving party through a safe dedicated line.

Specifically, fig. 5 is a schematic diagram of interaction between devices in the system provided in this specification, and in the data synchronization system in this specification, the local database 300 of the data sending party may first determine data to be transmitted, which is used as a data set to be synchronized.

Then, the local database 300 may determine a transmission task according to the data set to be synchronized, and create a transmission instance corresponding to the receiving database 306 in the transmission task for each receiving database 306 of the data receiving party, where the transmission instance carries the identifier of the receiving database 306.

Then, the local database 300 may determine the data package to be transmitted corresponding to the transmission instance from the data set to be synchronized according to the preset data receiving rule of the receiving database 306.

Finally, the transmission instance corresponding to the receiving database 306 may be executed, and the data package to be transmitted corresponding to the transmission instance and the identifier of the receiving database 306 are uploaded to the sending server 302.

The specific process of determining the data of each transmission instance and uploading the data of each transmission instance to the sending server 302 by the local database 300 has been described in detail in the above step S100 to step S106, and this specification is not described herein again, and reference may be made to the above content.

As shown in fig. 5, after the sending server 302 of the data sender receives the data of each transmission instance uploaded by the local database 300, the data of each transmission instance can be sent to the receiving server 304 through the dedicated security line. The data of the transmission instance at least includes the data package to be transmitted corresponding to the transmission instance and the identifier of the receiving database 306 corresponding to the transmission instance.

After the receiving server 304 receives the data of each transmission instance, for the data packet to be transmitted corresponding to each transmission instance, the receiving server 304 may write the data packet to be transmitted into the corresponding receiving database 306 according to the identifier of the receiving database 306 corresponding to the transmission instance.

The detailed process of the receiving server 304 writing the received data of each transmission instance into each receiving database 306 is already described in detail in the above step S200 to step S202, and this description is not repeated here, and the details can be referred to the above.

In the data synchronization system shown in this specification, the system may further include a plurality of local databases 300 and synchronization task servers corresponding to the plurality of local databases 300, as shown in fig. 6, a data set to be synchronized in each local database 300 may be determined and transmitted respectively, where the data set to be synchronized is determined in the local database 300.

Based on the data synchronization system shown in fig. 4, a local database in the system may first determine a data set to be synchronized, and for each receiving database, create a transmission instance corresponding to the receiving database. And then, according to a preset data receiving rule of the receiving database, determining a data package to be transmitted corresponding to the transmission example from the data set to be synchronized, and uploading the data package to be transmitted corresponding to the transmission example and the identification of the receiving database to a sending server by executing the transmission example corresponding to the receiving database. And then, the sending server in the system sends the data packet to be transmitted corresponding to each transmission instance and the identification of the receiving database corresponding to the data packet to be transmitted to the receiving server in the system through a safe special line. Finally, the receiving server can write the data packet to be transmitted into the corresponding receiving database according to the identifier of the receiving database corresponding to the transmission example aiming at the data packet to be transmitted corresponding to each transmission example. By determining the transmission examples corresponding to the receiving databases and writing the data packets to be transmitted corresponding to the transmission examples into the receiving databases corresponding to the transmission examples, the receiving databases do not need to poll to acquire data, and the timeliness of data transmission is high. Meanwhile, each receiving database is prevented from acquiring data which is not required by the receiving database, and the efficiency of data synchronization is improved.

Based on the data synchronization method shown in fig. 2, an embodiment of the present specification further provides a schematic structural diagram of a data synchronization apparatus, as shown in fig. 7.

Fig. 7 is a schematic structural diagram of a data synchronization apparatus provided in an embodiment of the present specification, where the apparatus includes:

a first determining module 400, which determines a data set to be synchronized through a local database;

a creating module 402, configured to create, for each receiving database, a transmission instance corresponding to the receiving database, where the transmission instance carries an identifier of the receiving database;

a second determining module 404, configured to determine, according to a preset data receiving rule of the receiving database, a to-be-transmitted data packet corresponding to the transmission instance from the to-be-synchronized data set;

the execution and write-in module 406 executes the transmission instance corresponding to the receiving database, uploads the to-be-transmitted data packet corresponding to the transmission instance and the identifier of the receiving database to the sending server, so that the sending server sends the to-be-transmitted data packet and the identifier of the receiving database to the receiving server through a secure dedicated line, and the receiving server writes the to-be-transmitted data packet into the receiving database according to the identifier of the receiving database.

Optionally, the creating module 402 is specifically configured to determine a transmission task according to the data set to be synchronized, and create, for each receiving database, a transmission instance corresponding to the receiving database included in the transmission task, where the transmission instance belongs to the transmission task.

Optionally, the second determining module 404 is specifically configured to determine, according to a stored correspondence between each receiving database and each data receiving rule, a data receiving rule of the receiving database, determine whether data matching the data receiving rule of the receiving database exists in the to-be-synchronized data set, if yes, use the matched data as the to-be-transmitted data packet corresponding to the transmission instance, and if not, determine that the transmission instance of the receiving database is executed completely.

Optionally, the executing and writing module 406 is specifically configured to, when the data packets to be transmitted corresponding to each transmission instance in the transmission task are determined, group the transmission instances according to the number of preset parallel instances, sequentially execute the transmission instances of each group through idle threads, respectively upload the data packets to be transmitted corresponding to the transmission instances of each group and the identifiers of the corresponding receiving databases to the sending server, and determine that the transmission task is completely executed until each group of transmission instances is completely executed.

Optionally, the executing and writing module 406 is specifically configured to, for each receiving database, determine a storage path of the receiving database preset in the sending server, and send the to-be-transmitted data packet corresponding to the transmission instance and the identifier of the receiving database to the storage path for storage, so that when the sending server monitors that data in the storage path changes, the to-be-transmitted data packet corresponding to the transmission instance and the identifier of the receiving database are sent to the receiving server.

Based on the data synchronization method shown in fig. 3, an embodiment of the present specification further provides a schematic structural diagram of a data synchronization apparatus, as shown in fig. 8.

Fig. 8 is a schematic structural diagram of a data synchronization apparatus provided in an embodiment of the present specification, where the apparatus includes:

the receiving module 500 is configured to receive, by a receiving server, data of each transmission instance sent by a sending server through a secure private line, where the data of the transmission instance at least includes a to-be-transmitted data packet corresponding to the transmission instance and an identifier of a receiving database corresponding to the transmission instance;

the writing module 502 writes, for a to-be-transmitted data packet corresponding to each transmission instance, the to-be-transmitted data packet into a corresponding receiving database according to an identifier of the receiving database corresponding to the transmission instance, where data of the transmission instance is uploaded from a local database to the sending server in the following manner: the local database creates a transmission example corresponding to each receiving database, determines a data packet to be transmitted corresponding to each transmission example from the data set to be synchronized based on the data receiving rule of each receiving database, and uploads the data of each transmission example to the sending server by executing each transmission example.

Optionally, the writing module 502 is specifically configured to determine an attribute of the receiving database according to the identifier of the receiving database corresponding to the transmission instance, where the attribute at least includes: and one of a data structure or a database structure, converting the format of the data packet to be transmitted according to the attribute of the receiving database, storing the converted data packet into the receiving database, and enabling the receiving database to perform service processing according to the converted data.

Optionally, the write-in module 502 is further configured to determine whether the data packet to be transmitted is data required for service processing according to a data file name in the data packet to be transmitted, write the data packet to be transmitted into a corresponding receiving database if the data packet to be transmitted is the data required for service processing, and not store the data packet to be transmitted if the data packet to be transmitted is the data required for service processing.

The present specification also provides a computer-readable storage medium storing a computer program which, when executed by a processor, is operable to perform the data synchronization method provided above.

Based on the data synchronization method provided above, the embodiment of the present specification further provides a schematic structural diagram of the electronic device shown in fig. 9. As shown in fig. 9, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, but may also include hardware required for other services. The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to realize the data synchronization method.

Of course, besides the software implementation, the present specification does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may be hardware or logic devices.

In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain the corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), is an integrated circuit whose Logic functions are determined by programming the Device by a user. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually making an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development and writing, but the original code before compiling is also written by a specific Programming Language, which is called Hardware Description Language (HDL), and HDL is not only one but many, such as abel (advanced Boolean Expression Language), ahdl (alternate Hardware Description Language), traffic, pl (core universal Programming Language), HDCal (jhdware Description Language), lang, Lola, HDL, laspam, hardward Description Language (vhr Description Language), vhal (Hardware Description Language), and vhigh-Language, which are currently used in most common. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may thus be considered a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.

The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functions of the various elements may be implemented in the same one or more software and/or hardware implementations of the present description.

As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The description has been presented with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the description. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

This description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only an example of the present specification, and is not intended to limit the present specification. Various modifications and alterations to this description will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification.

Claims

1. A method of data synchronization, comprising:

the local database determines a data set to be synchronized;

2. The method of claim 1, wherein creating, for each receiving database, a transmission instance corresponding to the receiving database specifically comprises:

determining a transmission task according to the data set to be synchronized;

3. The method according to claim 1 or 2, wherein determining the to-be-transmitted data package corresponding to the transmission instance from the to-be-synchronized data set according to a preset data receiving rule of the receiving database specifically includes:

4. The method of claim 3, wherein executing the transmission instance corresponding to the receiving database, and uploading the to-be-transmitted data package corresponding to the transmission instance and the identifier of the receiving database to a sending server, specifically comprises:

5. The method of claim 1, wherein uploading to a sending server specifically comprises:

6. A method of data synchronization, comprising:

7. The method of claim 6, wherein writing the data packet to be transmitted into the corresponding receiving database according to the identifier of the receiving database corresponding to the transmission instance, specifically comprises:

8. The method of claim 6, wherein the method further comprises:

and if not, not storing the data packet to be transmitted.

9. A data synchronization system, the system comprising: local database, sending server and receiving server, wherein:

10. A data synchronization apparatus, comprising:

11. A data synchronization device is characterized by specifically comprising:

12. A computer-readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method of any of the preceding claims 1-5 or 6-8.

13. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1-5 or 6-8 when executing the program.