CN112214547A

CN112214547A - Data processing method, data server, electronic device and storage medium

Info

Publication number: CN112214547A
Application number: CN202011016649.4A
Authority: CN
Inventors: 刘少耿; 刘青; 马根明
Original assignee: Baidu International Technology Shenzhen Co ltd
Current assignee: Baidu International Technology Shenzhen Co Ltd
Priority date: 2020-09-24
Filing date: 2020-09-24
Publication date: 2021-01-12

Abstract

The application discloses a data processing method, a data server, electronic equipment and a storage medium, and relates to the technical field of big data. The specific implementation scheme is as follows: when detecting that data in a data table of a source database changes, recording a timestamp of the changed data in the changed data; under the condition that the data table is subscribed, generating a data synchronization message based on the identification of the data table, the operation information of the changed data in the data table and the changed data; and adding the data synchronization message to a message queue of a message processing device so that the target data server subscribed to the data table synchronizes the changed data. According to the technical scheme, the data synchronization effect of the target data server side and the source data server side can be effectively improved, and the consistency of the data of the target data server side and the source data server side is guaranteed.

Description

Data processing method, data server, electronic device and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data processing method, a data server, an electronic device, and a storage medium.

Background

In practical application, data in one set of database is often required to be shared at different service ends, and the data sharing mode is realized in a data synchronization mode, that is, the same set of data can be stored with one copy of data at multiple ends.

In the prior art, data synchronization usually can transmit data from a source data service side to a target data service side for synchronization in a message queue manner, so that data of both sides can be decoupled, and services of both sides can be separated. For example, in a specific implementation, a source data service side may write a related message of data into a message queue, and the message queue may sequentially push a related message of data in the message queue to a target data service side subscribing to a data table to which the data belongs in a push manner based on a first-in first-out principle, so that the target data service side synchronizes the data based on the related message of the data.

However, in the above mode, the time sequence of the messages in the message queue can be guaranteed, but for the distributed system, the sequence of the messages entering the message queue from the source data service side cannot be guaranteed, and the corresponding sequence is the service operation sequence of the source data service side, so that the data of the target data service side is inconsistent with the data of the source data service side.

Disclosure of Invention

In order to solve the above problems, the present application provides a data processing method, a data server, an electronic device, and a storage medium.

According to an aspect of the present application, there is provided a data processing method, wherein the method includes:

when detecting that data in a data table of a source database changes, recording a timestamp of the changed data in the changed data;

under the condition that the data table is subscribed, generating a data synchronization message based on the identification of the data table, the operation information of the changed data in the data table and the changed data;

and adding the data synchronization message to a message queue of a message processing device so that the target data server subscribed to the data table synchronizes the changed data.

According to another aspect of the present application, there is provided a data processing method, wherein the method includes:

acquiring a data synchronization message from a message queue of a message processing device, wherein the data synchronization message comprises data which changes when the data changes, and a timestamp of the data change is recorded in the changed data;

and performing data synchronization processing based on the timestamp of the data change.

According to still another aspect of the present application, there is provided a source data server, wherein the source data server includes:

the recording module is used for recording a timestamp of data change in the changed data when detecting that the data in the data table of the source database changes;

the generation module is used for generating a data synchronization message based on the identification of the data table, the operation information of the changed data in the data table and the changed data under the condition that the data table is subscribed;

and the adding module is used for adding the data synchronization message to a message queue of a message processing device so as to synchronize the changed data of the target data server subscribed to the data table.

According to yet another aspect of the present application, there is provided a target data server, wherein the target data server includes:

the device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a data synchronization message from a message queue of a message processing device, the data synchronization message comprises data which changes when the data changes, and a timestamp of the data change is recorded in the changed data;

and the synchronous processing module is used for carrying out data synchronous processing based on the timestamp of the data change.

According to yet another aspect of the present application, there is provided an electronic device including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described above.

According to yet another aspect of the present application, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method as described above.

According to the technology of the application, the timestamp of the changed data is recorded in the changed data, and the timestamp can uniquely identify the service operation time of the source data server side so that a target data server subscribing a corresponding data table can perform data synchronization processing based on the timestamp of the changed data, the data synchronization effect of the target data server side and the source data server side can be effectively improved, and the consistency of the data of the target data server side and the source data server side is ensured.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:

FIG. 1 is an application architecture diagram of an embodiment of the present application;

FIG. 2 is a schematic diagram according to a first embodiment of the present application;

FIG. 3 is a schematic diagram according to a second embodiment of the present application;

FIG. 4 is a schematic illustration according to a third embodiment of the present application;

FIG. 5 is a schematic illustration according to a fourth embodiment of the present application;

FIG. 6 is a schematic illustration according to a fifth embodiment of the present application;

FIG. 7 is a schematic illustration according to a sixth embodiment of the present application;

FIG. 8 is a schematic illustration according to a seventh embodiment of the present application;

FIG. 9 is a schematic illustration according to an eighth embodiment of the present application;

FIG. 10 is a schematic illustration according to a ninth embodiment of the present application;

fig. 11 is a block diagram of an electronic device for implementing the data processing method according to the embodiment of the present application.

Detailed Description

The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is an application architecture diagram of an embodiment of the present application. As shown in fig. 1, data in a DataBase (DataBase; DB) on the source data server side may be shared by a plurality of target data service sides, and as shown in fig. 1, the data may include a target data server 1, a target data server 2, and a target data server …. Under the framework, the principle of data sharing is as follows: the source data server monitors the DB of the source data server side, when data synchronization is needed, the data synchronization message is written into a message queue of the message processing device, and when each target data server needs to synchronize data, the data synchronization message is obtained from the message queue, and the corresponding DB of the target data server side is updated according to the data synchronization message, so that the data synchronization with the DB of the source data server side is realized. By adopting the architecture shown in fig. 1, data at the source data server side and data at each target data server side can be decoupled, and services at the source data server side and the target data servers can be separated.

It should be noted that the source data server side may manage and maintain numerous data tables, and each target server may subscribe the data table required by itself in the message processing apparatus to obtain a message for synchronizing data related to the data table subscribed in the message queue, so as to implement synchronization processing on data in the subscribed data table.

FIG. 2 is a schematic diagram according to a first embodiment of the present application; as shown in fig. 2, this embodiment provides a data processing method, which may specifically include the following steps:

s101, when detecting that data in a data table of a source database changes, recording a timestamp of the data change in the changed data;

s102, under the condition that the data table is subscribed, generating a data synchronization message based on the identification of the data table, the operation information of the changed data in the data table and the changed data;

s103, adding a data synchronization message to a message queue of the message processing device so as to enable a target data server subscribing the data table to synchronize changed data.

The data processing method of this embodiment is described on the source data server side in the architecture diagram shown in fig. 1.

The source database in this embodiment refers to a database on the server side of the source data, and multiple data tables can be managed and maintained in the source database. When the source data server detects that the data in the data table of the source database is inserted, modified or deleted, the data in the corresponding data table is changed. At this time, the source data server may synchronously record operation information of the changed data. In addition, in this embodiment, it is necessary to record a time stamp indicating that data has changed in the changed data. Alternatively, the unit of the time stamp of the present embodiment may reach the millisecond or even microsecond level to accurately record the time when the data changes. Then, based on the identification of the data table, the operation information of the changed data in the data table, and the changed data, a data synchronization message is generated, that is, what operation occurred in the data in which data table is recorded in the data synchronization message, what the data after the operation is, and what the timestamp of the change occurred in the data due to the operation is. The data synchronization message is then added to the message queue of the message-out device. According to the method of this embodiment, since the changed data has the timestamp of the data change recorded therein, and the timestamp can uniquely identify the time of the service operation at the source data server side, when the same target server obtains multiple data synchronization messages, the sequence of the operation of the data change can be determined based on the timestamp of the data change in the multiple changed data, at this time, the identification information of the data synchronization message can be removed from the message queue, and when the target data server subsequently subscribed to the data table synchronizes the changed data, the data synchronization can be accurately completed according to other messages in the data synchronization message based on the timestamp of the changed data.

In addition, optionally, in this embodiment, when detecting that data in the data table of the source database changes, the source data server records a timestamp of the changed data in the changed data, and when implementing the method specifically, may update information of the time information field in the data based on the timestamp of the changed data.

That is to say, in this embodiment, a time information field may be added to the data table of the source database to record a timestamp of a change of corresponding data. And when the data in the data table changes every time, the time information field corresponding to the data is updated, so that the timestamp of the latest data change is always recorded in the time information field.

Optionally, in this embodiment, before generating the data synchronization message, it may also be detected whether a data table to which the changed data belongs is subscribed, and if so, the data synchronization message needs to be generated at this time for the subscribed target data server to update; otherwise, if the data table is not subscribed, the corresponding data synchronization message does not need to be generated at this time, because no target data server needs to update the data table.

In practical application, the target data server subscribes to a data table in the database of the source data server from the message processing device, and a corresponding subscription relationship between a data table identifier and a target data server identifier can be established in the message processing device based on a subscription request at the target data server side. Further, the message processing apparatus may obtain the identifications of all subscribed data tables based on the subscription relationship of all target data servers, and send the identifications to the source data server, so that the source data server records which data tables are subscribed. In this way, when detecting that the data in the subscribed data table changes, generating a corresponding data synchronization message; otherwise, the generation is not needed, and the realization is very flexible.

For example, according to the existing processing method, on the source data server side, if the business operation 1 is executed first, but the execution flow of the business operation 1 is more, and other related operations are not executed after the data change, while the initiation of the business operation 2 is performed after the business operation 1, but the operation flow is simple, and the business operation is executed soon, so when writing into the message queue, the data synchronization message corresponding to the business operation 2 is written first, and the data synchronization message of the business operation 1 is written later, and the queue follows the first-in first-out rule, thereby causing the sequence of the messages written into the message queue to be inconsistent with the real business operation sequence. If the business operations 1 and 2 relate to processing the same piece of data, on the target data server side, the target data server side does not execute the business operations according to the business operation sequence of the source data server side based on the sequence of executing the business operation 2 first and then executing the business operation 1, and then the data of the target data server side and the data of the source data server side are inconsistent.

By adopting the technical scheme of the embodiment, the message identifier is removed from the message queue, and the timestamp of the data change is added to the changed data corresponding to the business operation 1 and the business operation 2 respectively. In this case, when the target data server side performs data synchronization, data synchronization processing is performed based on the time stamp of the data change. Even if the business operation 1 and the business operation 2 correspond to the same data processing, the data synchronization processing is performed according to the timestamp of the changed data, if the business operation with the later timestamp is executed first, the business operation with the earlier timestamp can be discarded, so as to ensure the absolute synchronization of the data on the target data server side and the data on the source data server side.

Through the above comparative analysis, it can be known that, in the existing scheme, after the data synchronization message is added to the incoming message queue, the message queue ensures the time sequence of the data synchronization message. For example, for a distributed system, the sequence of entering the queue is often not guaranteed to be the sequence of the service end operation, for example, a service party performs 2 service operations in sequence on the same data, a service operation 1 updates its field status to 1, a service operation 2 updates its status to 2, and the theoretical sequence is that the service operation 1 enters the queue first and the service operation 2 enters the queue later, but due to the problem of resource scheduling of the distributed system, the actual sequence may be that the operation 2 enters the queue first and the service operation 1 enters the queue later, so that the receiving party will first update the service operation 2, i.e., status, to 2 and then the service operation 1, i.e., status, to 1 according to the queue sequence, which results in the problem of data inconsistency between the source data server and the target data server. The scheme of the embodiment can avoid the problem that the data timing is ensured by the service side instead of the identification d of the data synchronization message of the message queue. For example, a time information field may be added to each source data table structure, which may be named as a data _ version field, and its attribute is set to datetime, which is in the order of milliseconds or microseconds, and is "timestamp update according to operation time". For example, although the receiving side is reached by the sequence of the service operation 2 and the service operation 1, the receiving side will execute the service operation 2 first, and when the service operation 1 comes, the receiving side finds that the data _ version field in the changed data is earlier and does not update, so that the data of the source data server and the data of the target data server are kept consistent.

In the data processing method of this embodiment, when detecting that data in a data table of a source database changes, a timestamp indicating that the data changes is recorded in the changed data; generating a message based on the identification of the data table, the operation information of the changed data in the data table and the changed data; and when the data table is subscribed, adding a message to a message queue of the message processing device so that a target data server subscribed to the data table can synchronize changed data. In this embodiment, by recording the timestamp of the changed data in the changed data, the timestamp can uniquely identify the service operation time of the source data server side, so that the target data server subscribing to the corresponding data table performs data synchronization processing based on the timestamp of the changed data, the data synchronization effect of the target data server side and the source data server side can be effectively improved, and the consistency of the data of the target data server side and the source data server side is ensured.

FIG. 3 is a schematic diagram according to a second embodiment of the present application; as shown in fig. 3, the data processing method of the present embodiment further describes the technical solution of the present application in more detail on the basis of the technical solution of the embodiment shown in fig. 2. As shown in fig. 3, the data processing method of this embodiment may specifically include the following steps:

s201, receiving a data synchronization request which is sent by a target data server and carries a data table identifier and a time period identifier;

s202, acquiring various pieces of data operation information in corresponding time periods and changed data in corresponding data tables;

s203, generating a corresponding data synchronization message group based on each piece of data operation information of the time period and the changed data in the corresponding data table; the data synchronization message group comprises at least one data synchronization message;

s204, adding a data synchronization message group into a message queue of the message processing device so that the target data server can acquire the data synchronization message group from the message queue, and synchronizing data in a time period and a data table based on the data synchronization message group in the message queue.

The data processing method in the embodiment shown in fig. 2 can ensure that the data on the source data server side can be synchronized to the subscribed target data server side in time when the data on the source data server side changes, so that the data on the target data server side can be synchronized with the data on the source data server side. However, in practical applications, there may also be an abnormality in the message queue or the message processing device where the message queue is located, so that the target data server cannot acquire the data synchronization message in the message queue, and the data synchronization processing is not performed in time.

Therefore, based on this scenario, in this embodiment, the target data server may also perform data synchronization detection periodically, and detect whether the maintained data table is consistent with the data table on the source data server side. At this time, the target data server side may directly access the source data server to request to detect and query data in a certain data table, and compare the data with data in a corresponding data table in the target data server, if the data are not synchronized, the target data server may initiate a data synchronization request to the source data server, and the data synchronization request carries an identifier of the data table and a time period identifier. Since the target data server is regularly checked and ensures that the data after each check has been synchronized. Therefore, the target data server is regularly detected, which can be understood as detecting the data in the time period of the last detection. The time period may be a determined time interval based on the last detection of the local detection, and may be identified by, for example, a start time and an end time, or may be divided into a plurality of time segments in advance, and the periodic detection is performed at the start time of each time segment, and at this time, the corresponding time period may also be identified by using the identification of the time segment.

Correspondingly, on the side of the source data server, the source data server may receive a data synchronization request carrying a data table identifier and a time period identifier sent by the target data server, acquire each piece of data operation information in a corresponding time period and changed data in a corresponding data table, and then generate a corresponding data synchronization message group based on each piece of data operation information in the time period and changed data in the corresponding data table, where the data synchronization message group includes at least one data synchronization message; each of the data synchronization messages is generated in the same way as the data synchronization message in the embodiment shown in fig. 1. The data synchronization message group includes all data synchronization messages in the time period. Then, a data synchronization message group is added to the message queue of the message processing apparatus, so that the target data server can obtain the data synchronization message group from the message queue, and synchronize data in the data table within the time period based on the data synchronization message group in the message queue, which is implemented in the same principle as the principle of synchronizing one data synchronization message in the embodiment shown in fig. 1. In a similar manner, data synchronization is performed on the target data server side based on each data synchronization message in the data synchronization message group, and data consistency between the data table on the target data server side and the data table on the source data server side can be achieved.

In this embodiment, the target data server performs synchronous detection by using the data table and the time period as the detection unit, and if it is detected that the maintained data tables are all inconsistent with the data of the source data server side in the same time period, at this time, a data synchronization request may be respectively initiated based on each data table, or a data synchronization request including a plurality of data table identifiers and time period identifiers may be initiated simultaneously. For each data table, a corresponding data synchronization message group can be generated, so that the target data server can perform synchronization processing on the data in the data table based on the data synchronization message group. Of course, the data synchronization messages of all the data tables may also be combined together to form a large data synchronization message group, and at this time, for each data synchronization message, a corresponding data table needs to be determined, and then a corresponding data synchronization operation needs to be performed. In practical application, the corresponding operation processing mode may be selected based on specific requirements, which is not limited herein.

In the prior art, a related scheme is also realized by adopting a timing task, the difference (diff) of data at the sides of the target data server and the source data server is polled and detected at regular intervals, inconsistent data is alarmed, and a maintainer is informed to carry out manual synchronization. However, in the existing scheme, data are inconsistent every time, manual synchronization is needed, manpower is consumed, and maintenance personnel are online for less than 24 hours, so that a certain period exists on the online when the data are inconsistent. By adopting the technical scheme of the embodiment, the target data server monitors the task at regular time, and when finding that diff exists between the data of the target data server and the data of the target data server, the target data server notifies the source data server, and the source data server synchronizes the different data again to finish the automation of data synchronization, so that the labor cost can be effectively reduced.

By adopting the technical scheme, the data processing method of the embodiment can automatically start data synchronization processing when the asynchronism is periodically detected, so that the data synchronization processing can be performed between the source data server and the target data server in time, and the data synchronization between the source data server and the target data server is ensured. Compared with the prior art, the cost of manually carrying out data synchronization processing can be effectively reduced, and the data consistency between the source data server and the target data server is ensured.

FIG. 4 is a schematic illustration according to a third embodiment of the present application; as shown in fig. 4, this embodiment also provides a data processing method, which may specifically include the following steps:

s301, acquiring a data synchronization message from a message queue of a message processing device, wherein the data synchronization message comprises data which changes when the data changes, and a timestamp of the data change is recorded in the changed data;

s302, data synchronization processing is carried out based on the time stamp of the data change.

Different from the above-described embodiment shown in fig. 2, the data processing method of the present embodiment describes the technical solution of the present application on any target data server side in the architecture diagram shown in fig. 1.

In this embodiment, the message queue of the message processing apparatus stores the data synchronization messages of all the data tables in the database of the source data server, but each target data server may share only one, two or more data tables. Therefore, when each target data server performs data synchronization processing, only the data synchronization message corresponding to the maintained data table needs to be acquired, and the data synchronization message of the data table which is irrelevant to the target data server does not need to be acquired. Therefore, before this step S301, each target data server needs to subscribe to the data synchronization message of the required data table from the message processing apparatus. For example, a corresponding subscription relationship between the data table identifier and the target data server identifier may be established in the message processing apparatus based on a subscription request on the target data server side. In this way, the message processing apparatus may push the corresponding data synchronization message to the target data server that subscribes to the data table corresponding to the data synchronization message based on the subscription relationship. Or based on the request of each target data server, the corresponding target data server pulls the data synchronization message corresponding to the subscribed data table from the message queue. Of course, before the pulling, the message processing apparatus needs to detect that the target data server has the authority to pull the data synchronization message based on the subscription relationship.

In this embodiment, when the target data server obtains the data synchronization message from the message queue of the message processing apparatus, the data synchronization message includes data that changes when the data changes, and a timestamp indicating that the data changes is recorded in the changed data; in this way, the timestamp of the data change can be further acquired from the data synchronization message; referring to the related description of the embodiment described in fig. 2, the timestamp of the data change can uniquely identify the source data server side, and the time sequence of the service operation, in this embodiment, the data synchronization process is performed by referring to other messages in the data synchronization message based on the timestamp of the data change, so that the consistency between the data of the target data server side and the data of the source data server side can be ensured.

For example, if multiple data synchronization messages are acquired simultaneously, the data synchronization messages may be sequentially executed based on the sequence of timestamps in which data in the data synchronization messages change, and it is determined that the sequence of data operations is consistent with the sequence of service operations of the source data server, so as to ensure the consistency of data.

In the data processing method of the embodiment, a data synchronization message is acquired from a message queue of a message processing device, the data synchronization message includes data changed when the data is changed, and a timestamp of the data change is recorded in the changed data; the data synchronization processing is carried out based on the timestamp of the data change, and the timestamp can uniquely identify the business operation time of the source data server side, and the target data server carries out the data synchronization processing based on the timestamp of the data change, so that the data synchronization effect of the target data server side and the source data server side can be effectively improved, and the consistency of the data of the target data server side and the source data server side is ensured.

FIG. 5 is a schematic illustration according to a fourth embodiment of the present application; as shown in fig. 5, the data processing method of the present embodiment further describes the technical solution of the present application in more detail on the basis of the technical solution of the embodiment shown in fig. 4. As shown in fig. 5, the data processing method of this embodiment may specifically include the following steps:

s401, acquiring a data synchronization message from a message queue of a message processing device, wherein the data synchronization message comprises an identifier of a data table, operation information of changed data in the data table and changed data when the data are changed, and a timestamp of the changed data is recorded in the changed data;

s402, detecting whether the processed message relates to the same data in the data synchronization message; if yes, go to step S403; otherwise, go to step S405;

s403, detecting whether the time stamp of the data change recorded in the data synchronization message is earlier than the time stamp of the data change recorded in the processed message; if yes, go to step S404; otherwise, go to step S405;

s404, discarding the data synchronization message to ensure the synchronization of the data, and ending.

S405, synchronizing the changed data in the data table based on the identification of the data table in the data synchronization message and the operation information of the changed data in the data table, and ending.

In this embodiment, it is taken as an example that one data synchronization message is acquired at a target data server side each time. However, in order to ensure the synchronization of the data and the source data server side, for each acquired data synchronization message, before performing data synchronization processing, it is detected whether the data synchronization message is the same as the processing related to the processed message, if so, it is detected whether a timestamp of a change of the data recorded in the data synchronization message is earlier than a timestamp of a change of the data recorded in the processed message, if so, it indicates that the data synchronization message should be received earlier and processed earlier, but now is later than the timestamp of the change of the data recorded in the processed message, and if so, the operation is executed, which may cause the operation sequence to be disordered, and at this time, the data synchronization message may be discarded without being executed, so as to ensure the synchronization of the data. And the timestamp of the data change recorded in the data synchronization message is later than the timestamp of the data change recorded in the processed message, or the data related to the data synchronization message and the data related to the processed message are not the same data, at this time, the data synchronization message and the processed message are not affected with each other, and the data changed in the data table is synchronized directly based on the identification of the data table in the data synchronization message and the operation information of the data changed in the data table.

By adopting the above technical solution, the data processing method of this embodiment can perform data synchronization processing based on the timestamp of the change of the data, and discard the data synchronization message when whether the timestamp of the change of the data recorded in the data synchronization message is earlier than the timestamp of the change of the data recorded in the processed message, so as to ensure the synchronization of the data; and when the processed message and the data synchronization message do not relate to the same data, or the timestamp of the data change recorded in the data synchronization message is later than the timestamp of the data change recorded in the processed message, synchronizing the changed data in the data table based on the identification of the data table in the data synchronization message and the operation information of the changed data in the data table, thereby effectively improving the data synchronization effect of the target data server side and the source data server side and ensuring the consistency of the data of the target data server side and the source data server side.

FIG. 6 is a schematic illustration according to a fifth embodiment of the present application; as shown in fig. 6, the data processing method of the present embodiment further describes the technical solution of the present application in more detail on the basis of the technical solutions of the embodiments shown in fig. 4 or fig. 5. As shown in fig. 6, the data processing method of this embodiment may specifically include the following steps:

s501, periodically detecting whether a data table in a time period is synchronous with a data table in a source database of a subscribed source data server side; if not, executing step S502; otherwise, returning to the step S501 to continue detection;

s502, sending a data synchronization request carrying a data table identifier and a time period identifier to a source data server;

s503, acquiring a data synchronization message group from the message queue, wherein the data synchronization message group is a message group which is generated by the source data server based on the data table identifier and the time period identifier and comprises at least one data synchronization message;

and S504, synchronizing the data in the data table in the time period in the database at the side of the target data server based on the data synchronization message group.

The above steps S501 to S504 differ from the above embodiment shown in fig. 3 only in that: the embodiment shown in fig. 3 above describes the technical solution of the present application on the source data server side; however, in this embodiment, the technical solution of the present application is described on the target data server side, and the remaining implementation principles are the same as those in the embodiment shown in fig. 3, and reference may be made to the description of the embodiment shown in fig. 3 for details, which is not described herein again.

And S505, when the new data table needs to be synchronized, generating a configuration file of the new data table based on the structure of the new data table and the synchronized data table and the logic function implementation file of the synchronized data table.

When a new data table needs to be synchronized on the target data server side, the structure of the new data table needs to be configured on the target data server side in advance, and then various logic functions of the new data table need to be configured on the target data server side, for example, various functions such as how to look up the data table, how to implement operations such as inserting, modifying or deleting data in the data table, and the like, may be included, so as to ensure that all functions of the data table can be used on the target data server side. In practical application, the logic functions of the data table are realized by writing program codes, completing testing and then connecting the data table to the line, and the process needs to consume certain labor cost to write the program codes and easily causes errors of old functions.

In order to overcome the technical problem, in the present application, a same set of program codes may be deployed, and when a new data table needs to be synchronized, based on the structure of the new data table and the synchronized data table and the logic function implementation file of the synchronized data table, only the changed portion of the structure of the new data table with respect to the structure of the synchronized data table is referred to the logic function implementation file of the synchronized data table, abstracted as a configuration file, and output in a json format, where:

wherein mirrorral _ ral and to _ ral represent service configurations of data in the synchronized data table and the new data table, respectively;

topic _ name represents the name of the synchronized database table;

table _ name represents the name of the new database table; the pattern is the table name of the new data table, the handler is the strategy of the sublist, and the field is the field of the sublist.

The allow _ topic _ types represent the type of data synchronization operation, and 1, 2 and 3 are insert, update and delete operations, respectively.

fields represents the fields of the database table, the name of each field is a field name, the type of each field is a field type, required represents whether a data source needs to transmit the fields, and primary _ key represents whether the primary key of the table is required.

Step S505 is a supplement scheme provided in this embodiment, and step S505 may also be combined with the embodiments shown in fig. 4 or fig. 5, respectively, to form an optional technical solution of the present application.

In the scheme of step S505, the configuration file of the new data table is abstracted, and the newly added data table does not need to be modified by program codes, so that the development efficiency can be effectively improved, the cost can be saved, and the development accuracy can be improved.

FIG. 7 is a schematic illustration according to a sixth embodiment of the present application; as shown in fig. 7, the present embodiment provides a source data server 700, where the source data server 700 includes:

a recording module 701, configured to record a timestamp indicating that data changes in changed data when it is detected that data in a data table of a source database changes;

a generating module 702, configured to generate a data synchronization message based on an identifier of the data table, operation information of changed data in the data table, and the changed data when the data table is subscribed;

an adding module 703 is configured to add a data synchronization message to a message queue of the message processing apparatus, so that a target data server subscribing to the data table synchronizes changed data.

The source data server 700 of this embodiment, which implements the implementation principle and technical effect of data processing by using the modules, is the same as the implementation of the related method embodiment, and reference may be made to the description of the related method embodiment in detail, which is not described herein again.

FIG. 8 is a schematic illustration according to a seventh embodiment of the present application; as shown in fig. 8, the source data server 700 of this embodiment further describes the technical solution of the present application in more detail based on the technical solution of the embodiment shown in fig. 7.

The source data server 700 of this embodiment, the recording module 701 is configured to:

when detecting that the data in the data table of the source database changes, updating the information of the time information field in the data based on the timestamp of the data change.

Further optionally, as shown in fig. 8, the source data server 700 of this embodiment further includes:

an adding module 704, configured to add a time information field in the data table of the source database.

a detection module 705 for detecting and determining that the data table is subscribed to.

Further optionally, as shown in fig. 8, the source data server 700 of this embodiment further includes a receiving module 706 and an obtaining module 707;

a receiving module 706, configured to receive a data synchronization request carrying a data table identifier and a time period identifier sent by a target data server;

an obtaining module 707, configured to obtain each piece of data operation information in a corresponding time period and changed data in a corresponding data table;

a generating module 702, configured to generate a corresponding data synchronization message group based on each piece of data operation information of the time period and the changed data in the corresponding data table, where the data synchronization message group includes at least one data synchronization message;

the adding module 703 is further configured to add a data synchronization message group to a message queue of the message processing apparatus, so that the target data server obtains the data synchronization message group from the message queue, and synchronizes data in the data table within the time period based on the data synchronization message group in the message queue.

FIG. 9 is a schematic illustration according to an eighth embodiment of the present application; as shown in fig. 8, the present embodiment provides a target data server 900, and the target data server 900 includes:

an obtaining module 901, configured to obtain a data synchronization message from a message queue of a message processing apparatus, where the data synchronization message includes data that changes when the data changes, and a timestamp indicating that the data changes is recorded in the changed data;

and a synchronization processing module 902, configured to perform data synchronization processing based on the timestamp of the data change.

The target data server 900 of this embodiment, which implements the implementation principle and technical effect of data processing by using the modules, is the same as the implementation of the related method embodiment, and reference may be made to the description of the related method embodiment in detail, which is not described herein again.

FIG. 10 is a schematic illustration according to a ninth embodiment of the present application; as shown in fig. 10, the target data server 900 of the present embodiment further describes the technical solution of the present application in more detail on the basis of the technical solution of the embodiment shown in fig. 9.

As shown in fig. 10, in the target data server 900 of the present embodiment, the synchronization processing module 902 includes:

an information detecting unit 9021, configured to detect whether the processed message relates to the same piece of data in the data synchronization message;

the information detection unit 9021 is further configured to detect whether a timestamp of a change of data recorded in the data synchronization message is earlier than a timestamp of a change of data recorded in the processed message if the data synchronization message is positive;

and the processing unit 9022 is configured to discard the data synchronization message if the data synchronization message is received, so as to ensure data synchronization.

Further optionally, the data synchronization message further includes an identifier of a data table to which the data belongs and operation information of the changed data in the data table; a processing unit 9022, further configured to:

and if the same data in the data synchronization message is not related in the processed message, or the timestamp of the changed data recorded in the data synchronization message is later than the timestamp of the changed data recorded in the processed message, carrying out synchronization processing on the changed data in the data table based on the identification of the data table and the operation information of the changed data in the data table.

Further optionally, as shown in fig. 10, the target data server 900 of this embodiment further includes a synchronization detection module 903 and a sending module 904:

a synchronization detection module 903, configured to periodically detect whether a data table in a time period is synchronized with a data table in a source database on the source data server side;

a sending module 904, configured to send a data synchronization request carrying a data table identifier and a time period identifier to the source data server if the data synchronization request is not synchronized;

the obtaining module 901 is further configured to obtain a data synchronization message group from the message queue, where the data synchronization message group is a message group that is generated by the source data server based on the data table identifier and the time period identifier and includes at least one data synchronization message;

the synchronization processing module 902 is further configured to synchronize data in the data table within the time period in the database on the target data server side based on the data synchronization message group.

For example, the processing unit 9022 may specifically synchronize data in the data table within the time period in the database on the target data server side based on the data synchronization message group.

Further optionally, as shown in fig. 10, the target data server 900 of this embodiment further includes:

the generating module 905 is configured to generate a configuration file of a new data table based on the structures of the new data table and the synchronized data table and the logic function implementation file of the synchronized data table when the new data table needs to be synchronized.

According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.

Fig. 11 is a block diagram of an electronic device implementing a data processing method according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein. The electronic device of this embodiment may act as a source data server or a target data server respectively,

as shown in fig. 11, the electronic apparatus includes: one or more processors 1101, a memory 1102, and interfaces for connecting the various components, including a high speed interface and a low speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). In fig. 11, a processor 1101 is taken as an example.

The memory 1102 is a non-transitory computer readable storage medium as provided herein. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the data processing method provided by the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to execute the data processing method provided by the present application.

The memory 1102, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the data processing method in the embodiments of the present application (e.g., the relevant modules shown in fig. 7 and 8, or fig. 9 and 10). The processor 1101 executes various functional applications of the server and data processing by running non-transitory software programs, instructions, and modules stored in the memory 1102, that is, implements the data processing method in the above-described method embodiment.

The memory 1102 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of an electronic device implementing the data processing method, and the like. Further, the memory 1102 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 1102 may optionally include memory located remotely from the processor 1101, which may be connected via a network to an electronic device implementing the data processing method. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device implementing the data processing method may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected by a bus or other means, and are exemplified by being connected by a bus in fig. 11.

The input device 1103 may receive input numeric or character information and generate key signal inputs related to user settings and function controls of an electronic apparatus implementing the data processing method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer, one or more mouse buttons, a track ball, a joystick, or other input devices. The output devices 1104 may include a display device, auxiliary lighting devices (e.g., LEDs), tactile feedback devices (e.g., vibrating motors), and the like. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), the internet, and blockchain networks.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

According to the technical scheme of the embodiment of the application, when the data in the data table of the source database is detected to be changed, the time stamp of the data change is recorded in the changed data; generating a message based on the identification of the data table, the operation information of the changed data in the data table and the changed data; and when the data table is subscribed, adding a message to a message queue of the message processing device so that a target data server subscribed to the data table can synchronize changed data. In this embodiment, by recording the timestamp of the changed data in the changed data, the timestamp can uniquely identify the service operation time of the source data server side, so that the target data server subscribing to the corresponding data table performs data synchronization processing based on the timestamp of the changed data, the data synchronization effect of the target data server side and the source data server side can be effectively improved, and the consistency of the data of the target data server side and the source data server side is ensured.

According to the technical scheme of the embodiment of the application, when the asynchronism is periodically detected, the data synchronization processing can be automatically started, so that the data synchronization processing can be timely carried out between the source data server and the target data server, and the data synchronization between the source data server and the target data server is ensured. Compared with the prior art, the cost of manually carrying out data synchronization processing can be effectively reduced, and the data consistency between the source data server and the target data server is ensured.

According to the technical scheme of the embodiment of the application, the data synchronization message is acquired from the message queue of the message processing device, the data synchronization message comprises changed data when the data are changed, and a timestamp of the data change is recorded in the changed data; the data synchronization processing is carried out based on the timestamp of the data change, and the timestamp can uniquely identify the business operation time of the source data server side, and the target data server carries out the data synchronization processing based on the timestamp of the data change, so that the data synchronization effect of the target data server side and the source data server side can be effectively improved, and the consistency of the data of the target data server side and the source data server side is ensured.

According to the technical scheme of the embodiment of the application, data synchronization processing can be performed based on the timestamp of the data change, and the data synchronization message can be discarded when whether the timestamp of the data change recorded in the data synchronization message is earlier than the timestamp of the data change recorded in the processed message, so that the data synchronization is ensured; and when the processed message and the data synchronization message do not relate to the same data, or the timestamp of the data change recorded in the data synchronization message is later than the timestamp of the data change recorded in the processed message, synchronizing the changed data in the data table based on the identification of the data table in the data synchronization message and the operation information of the changed data in the data table, thereby effectively improving the data synchronization effect of the target data server side and the source data server side and ensuring the consistency of the data of the target data server side and the source data server side.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.

The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of data processing, wherein the method comprises:

2. The method of claim 1, wherein recording a timestamp of a change in data in changed data when a change in data in a data table of a source database is detected comprises:

when detecting that the data in the data table of the source database changes, updating the information of the time information field in the data based on the timestamp of the change of the data.

3. The method of claim 2, wherein, upon detecting a change in data in the data table of the source database, prior to updating information of a time information field in the data based on a timestamp of the change in the data, the method further comprises:

adding the time information field in the data table of the source database.

4. The method of claim 1, wherein before generating the data synchronization message based on the identification of the data table, the operational information of the data that has changed in the data table, and the changed data, the method further comprises:

detecting and determining that the data table is subscribed to.

5. The method of any of claims 1-4, wherein the method further comprises:

receiving a data synchronization request which is sent by the target data server and carries the data table identification and the time period identification;

acquiring each piece of data operation information in the corresponding time period and the changed data in the corresponding data table;

generating a corresponding data synchronization message group based on each piece of data operation information of the time period and the changed data in the corresponding data table, wherein the data synchronization message group comprises at least one piece of data synchronization message;

and adding the data synchronization message group into a message queue of the message processing device so that the target data server can acquire the data synchronization message group from the message queue, and synchronizing the data in the data table in the time period based on the data synchronization message group in the message queue.

6. A method of data processing, wherein the method comprises:

7. The method of claim 6, wherein performing data synchronization processing based on the timestamp of the change in the data comprises:

detecting whether the same piece of data in the data synchronization message is involved in the processed message;

if so, detecting whether the timestamp of the data change recorded in the data synchronization message is earlier than the timestamp of the data change recorded in the processed message;

and if so, discarding the data synchronization message to ensure the synchronization of the data.

8. The method according to claim 7, wherein the data synchronization message further includes an identification of a data table to which the data belongs and operation information of the data in which the change occurs; the method further comprises the following steps:

if the same data in the data synchronization message is not involved in the processed message, or the timestamp of the changed data recorded in the data synchronization message is later than the timestamp of the changed data recorded in the processed message, the changed data in the data table is synchronized based on the identification of the data table and the operation information of the changed data in the data table.

9. The method according to any one of claims 6-8, further comprising:

periodically detecting whether a data table in a time period is synchronous with a data table in a source database of a subscribed source data server side;

if not, sending a data synchronization request carrying a data table identifier and a time period identifier to the source data server;

acquiring a data synchronization message group from the message queue, wherein the data synchronization message group is a message group which is generated by the source data server based on the data table identifier and the time period identifier and comprises at least one data synchronization message;

and synchronizing the data in the data table in the time period in a database at the target data server side based on the data synchronization message group.

10. The method according to any one of claims 6-8, further comprising:

and when a new data table needs to be synchronized, generating a configuration file of the new data table based on the structure of the new data table and the synchronized data table and the logic function implementation file of the synchronized data table.

11. A source data server, wherein the source data server comprises:

12. The source data server of claim 11, wherein the logging module is to:

13. The source data server of claim 12, wherein the source data server further comprises:

and the adding module is used for adding the time information field in the data table of the source database.

14. The origin data server of claim 11, wherein the origin data server further comprises:

and the detection module is used for detecting and determining that the data table is subscribed.

15. The source data server of any of claims 11-14, wherein the source data server further comprises a receiving module and an obtaining module:

the receiving module is used for receiving a data synchronization request which is sent by the target data server and carries the data table identifier and the time period identifier;

the acquisition module is used for acquiring each piece of data operation information in the corresponding time period and the changed data in the corresponding data table;

the generating module is further configured to generate a corresponding data synchronization message group based on each piece of the data operation information of the time period and the changed data in the corresponding data table, where the data synchronization message group includes at least one data synchronization message;

the adding module is further configured to add the data synchronization message group to a message queue of the message processing apparatus, so that the target data server obtains the data synchronization message group from the message queue, and synchronizes data in the data table within the time period based on the data synchronization message group in the message queue.

16. A target data server, wherein the target data server comprises:

17. The target data server of claim 16, wherein the synchronization processing module comprises:

the information detection unit is used for detecting whether the same piece of data in the data synchronization message is related to the processed message or not;

the information detection unit is further configured to detect whether a timestamp of the change of the data recorded in the data synchronization message is earlier than a timestamp of the change of the data recorded in the processed message if the change of the data recorded in the data synchronization message is positive;

and the processing unit is used for discarding the data synchronization message if the data synchronization message is valid so as to ensure the data synchronization.

18. The target data server of claim 17, wherein the data synchronization message further includes an identification of a data table to which the data belongs and operation information of the data in which the change occurs; the processing unit is further configured to:

19. The target data server of any of claims 16-18, further comprising a synchronization detection module and a sending module:

the synchronous detection module is used for periodically detecting whether a data table in a time period is synchronous with a data table in a source database at the side of a subscribed source data server;

the sending module is used for sending a data synchronization request carrying a data table identifier and a time period identifier to the source data server if the source data server is not synchronized;

the obtaining module is further configured to obtain a data synchronization message group from the message queue, where the data synchronization message group is a message group that is generated by the source data server based on the data table identifier and the time period identifier and includes at least one data synchronization message;

and the synchronization processing module is further configured to synchronize data in the data table in the time period in a database on the side of the target data server based on the data synchronization message group.

20. The target data server of any of claims 16-18, the target data server further comprising:

and the generating module is used for generating a configuration file of the new data table based on the structures of the new data table and the synchronized data table and the logic function implementation file of the synchronized data table when the new data table needs to be synchronized.

21. An electronic device, comprising:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5, or 6-10.

22. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any of claims 1-5, or 6-10.