WO2020173381A1

WO2020173381A1 - Data interworking method and device, terminal and storage medium

Info

Publication number: WO2020173381A1
Application number: PCT/CN2020/076050
Authority: WO
Inventors: Weijie SONG (宋维捷)
Original assignee: 北京字节跳动网络技术有限公司
Priority date: 2019-02-27
Filing date: 2020-02-20
Publication date: 2020-09-03
Also published as: CN110083657A

Abstract

Disclosed in embodiments of the present disclosure are a data interworking method and device, a terminal and a storage medium. The data interworking method comprises: a second database receiving synchronization data transmitted in real time by a first database by means of a first data pipe (TPIPE) assembly; and controlling the second database to transmit, to a slave database of the second database, and by means of a second data pipe (TPIPE) assembly, full data in the second database, wherein the first data pipe (TPIPE) assembly is used to orderly transmit the synchronization data, and the second data pipe (TPIPE) assembly is used to orderly transmit the full data.

Description

Data intercommunication method, device, terminal and storage medium

This application claims the priority of the Chinese patent application filed with the Chinese Patent Office with application number 201910146908.6 on February 27, 2019. The entire content of this application is incorporated into this application by reference. Technical field

The embodiments of the present disclosure relate to the field of computer application technology, for example, to a data intercommunication method, device, terminal, and storage medium. Background technique

Remote Dictionary Server (Redis) is a non-relational memory database that is widely used in Nosql databases. Inside Redis is a key-value storage system. Because Redis has the advantages of easy expansion, large data volume to improve performance, and diverse and flexible data models, it is widely used.

Living in different places has become a data management solution adopted by more and more large Internet companies, and it is almost an inevitable choice for large-scale applications to develop to a certain stage. Multi-activity in different places generally refers to the establishment of independent data centers in different cities. These computer rooms also need to carry traffic in daily business to provide business support. That is, data exchange is performed between the databases of two computer rooms in different cities, so that users can access data in one place in different cities.

Take the data intercommunication between two Redis databases as an example. The Redis database data intercommunication method of related technologies usually directly transfers the data between the two Redis databases. Because the Redis database may include historical data and write in real time Incremental data makes it extremely prone to errors during data transmission, and it is often difficult to achieve data intercommunication without affecting the data, and the user experience is poor. Summary of the invention

The embodiments of the present disclosure provide a data intercommunication method, device, terminal, and storage medium to solve technical problems such as error-proneness and poor user experience when directly transmitting data between a database and a database in related technologies, and can achieve data intercommunication simply and effectively.

The embodiment of the present disclosure provides a data intercommunication method, which includes:

Receiving, through the second database, the data to be synchronized that the first database transmits in real time through the first data pipeline TPIPE component;

Controlling the second database to transmit the full amount of data in the second database to the secondary database of the second database through the second data pipeline TPIPE component; Wherein, the data to be synchronized includes historical data of the first database and first incremental data in the first database, and the full amount of data includes the data to be synchronized transmitted by the first database and the second The local end writes data in the database, the first data pipe TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipe TPIPE component is used to orderly transmit the full amount of data.

The embodiment of the present disclosure also provides a data intercommunication method, which includes:

Controlling the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component;

Controlling the first data pipeline TPIPE component to transmit the to-be-synchronized data of the first database to the second database in real time;

Wherein, the data to be synchronized includes historical data of the first database and first incremental data in the first database, and the first data pipeline TPIPE component is used for orderly transmitting the data to be synchronized.

The embodiment of the present disclosure also provides a data intercommunication device, which includes:

A data receiving module, configured to receive, through the second database, the data to be synchronized that the first database transmits in real time through the TPIPE component of the first data pipeline;

A full data transmission module, configured to control the second database to transmit the full data in the second database to the second database through the second data pipeline TPIPE component;

Wherein, the data to be synchronized includes historical data of the first database and first incremental data in the first database, and the full amount of data includes the data to be synchronized transmitted by the first database and the second database The local end writes data in, the first data pipe TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipe TPIPE component is used to orderly transmit the full amount of data.

The first data transmission module is configured to control the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component;

The second data transmission module is configured to control the first data pipeline TPIPE component to transmit the to-be-synchronized data of the first database to the second database in real time;

Wherein, the data to be synchronized includes historical data of the first database and first incremental data in the first database. The first data pipeline TPIPE component is used to orderly transmit the data to be synchronized.

The embodiments of the present disclosure also provide an electronic device, which includes:

One or more processing devices;

Memory, set to store one or more programs; When the one or more programs are executed by the one or more processing apparatuses, the one or more processing apparatuses implement the data intercommunication method according to any embodiment of the present disclosure.

The embodiments of the present disclosure also provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the data intercommunication method as described in any embodiment of the present disclosure is implemented. Description of the drawings

Fig. 1 is a schematic flowchart of a data interworking method provided in the first embodiment of the present disclosure; Fig. 2 is a schematic flowchart of a data interworking method provided in the second embodiment of the present disclosure; Fig. 4 is a schematic flowchart of a data intercommunication method provided by Embodiment 4 of the present disclosure; Fig. 5a is a schematic flowchart of a data intercommunication system provided by Embodiment 5 of the present disclosure; 5b is a schematic diagram of one-way data synchronization provided in the fifth embodiment of the present disclosure;

Figure 5c is a schematic diagram of the interaction of two-way data synchronization provided in the fifth embodiment of the present disclosure; Figure 5d is another schematic diagram of the interaction of two-way data synchronization provided in the fifth embodiment of the present disclosure; Figure 5e is another schematic diagram of the two-way data synchronization provided in the fifth embodiment of the present disclosure; A schematic diagram of the interaction of two-way data synchronization; Figure 5f is another schematic diagram of the interaction of two-way data synchronization provided by the fifth embodiment of the present disclosure; Figure 5g is another schematic diagram of the interaction of two-way data synchronization provided by the fifth embodiment of the present disclosure; 5h is another schematic diagram of the interaction of two-way data synchronization provided by the fifth embodiment of the present disclosure; FIG. 5i is a schematic diagram of the data interaction for data intercommunication provided by the fifth embodiment of the present disclosure; FIG. 6 is a schematic diagram of the data interaction provided by the sixth embodiment of the present disclosure Fig. 7 is a schematic structural diagram of a data intercommunication device provided in the seventh embodiment of the present disclosure; Fig. 8 is a schematic structural diagram of an electronic device provided in the eighth embodiment of the present disclosure. DETAILED DESCRIPTION The present disclosure will be described below with reference to the drawings and embodiments. It can be understood that the specific embodiments described here are only used to explain the present disclosure, but not to limit the present disclosure. In addition, it should be noted that, for ease of description, the drawings only show a part of the structure related to the present disclosure, but not all of the structure.

Example one

FIG. 1 is a schematic flowchart of a data intercommunication method provided in Embodiment 1 of the present disclosure. This example is suitable for data intercommunication between two independent databases, which can be used to realize multiple activities in different places. This method can be executed by a data intercommunication device, and the device can be configured in a terminal or a server to execute the data intercommunication method of this embodiment. In this embodiment, whether the two databases are in different places can be determined according to the round-trip time (Round-Trip Time, RTT) between the two databases. That is, according to the total time delay experienced from the sending end of the data transmission until the sending end receives the confirmation from the receiving end, the receiving end generally sends the confirmation immediately after receiving the data.

As shown in FIG. 1, the method of this embodiment may include the following steps:

S110: Receive, through the second database, the data to be synchronized that is transmitted in real time by the first database through the TPIPE component of the first data pipeline.

In this embodiment, the data to be synchronized includes historical data in the first database and first incremental data in the first database. The first data pipeline TPIPE component is used to orderly transmit the to-be-synchronized data of the first database.

In this embodiment, the historical data and the first incremental data can be divided according to actual needs. For example, the historical data may be data stored at the current time in the first database; the first incremental data may be data written after the current time. The current moment may be the moment when the first database starts to transmit the data to be synchronized in real time through the first data pipeline TPIPE component.

A large amount of to-be-synchronized data transmitted in real time by the first database through the TPIPE component of the first data pipeline is received through the second database, that is, one-way synchronization between the second database and the first database is established through the TPIPE component of the first data pipeline.

In the embodiment of the present disclosure, the first database and the second database may include non-relational memory databases. Exemplarily, the first database and the second database may both be Redis databases.

S120. Control the second database to transmit the full amount of data in the second database to the secondary database of the second database through the second data pipeline TPIPE component.

In this embodiment, the full amount of data includes the data to be synchronized transmitted by the first database and the local write data in the second database. As described above, the second database has received the data to be synchronized from the first database. , The second database includes both the to-be-synchronized data transmitted by the first database and the locally written data in the second database.

In the embodiment of the present disclosure, the first data pipe TPIPE component and the second data pipe TPIPE component may be message queues for orderly transmission of data. The second data pipe TPIPE component can be used to orderly transmit the full amount of data. In an embodiment, before the second database transmits the full amount of data to the secondary database of the second database through the second data pipeline TPIPE component, the method includes: establishing the secondary database of the second database. In this embodiment, the slave database can replace the data access of the first database, and provide data in the first database and the second database at the same time, so as to realize the The data of the first database and the second database are interconnected.

In this embodiment, the first database may also be called the first main database, and the second database may also be called the second main database.

In the technical solution of this embodiment, the second database includes the data written by the local end and the data to be synchronized transmitted by the first database received through the first data pipeline TPIPE component, so that users of the second database can access the first database at the same time And the data of the second database; and then transfer the full amount of data in the second database to the secondary database of the second database, so that users who originally accessed the first database can simultaneously access the first database and the second database through the secondary database of the second database Database data realizes data intercommunication. Moreover, the orderly transmission of data can be realized through the first data pipeline TPIPE component and the second data pipeline TPIPE component, which can ensure the integrity and real-time performance of data transmission, and solve the removal of data intercommunication methods. The technical problems of storage and poor user experience can neither affect the transmission of historical data nor the transmission of incremental data without stopping the machine, and truly realize the data intercommunication of the database.

Example two

FIG. 2 is a schematic flowchart of a data intercommunication method provided in Embodiment 2 of the present disclosure. This embodiment is described on the basis of the foregoing embodiment. In this embodiment, receiving, through the second database, the data to be synchronized that is transmitted in real time by the first database through the first data pipeline TPIPE component includes: receiving through the first dual-write component of the second database when the first synchronization condition is satisfied The first database transmits the data to be synchronized in real time through the first data pipeline TPIPE component, and controls the second database to receive the data to be synchronized in real time transmitted by the first database written by the first dual-write component of the second database.

In an embodiment, the data intercommunication method of this embodiment may further include: in a case where a second synchronization condition is met, controlling the second database to write the slave database to the slave database through the first dual-write component The second incremental data in the second database.

On this basis, the data intercommunication method of this embodiment may further include: controlling the second database to receive the third increment of the slave database written by the slave database through the second double-write component of the slave database data.

As shown in FIG. 2, the method of this embodiment may include the following steps:

S210: When the first synchronization condition is met, receive the data to be synchronized that the first database transmits in real time through the first data pipeline TPIPE component through the first dual-write component of the second database, and control the second database to receive the second database. The data to be synchronized for real-time transmission of the first database written by the first dual-write component of the second database.

S220: Control the second database to transmit all the data in the second database to the second database through the second data pipeline TPIPE component. In this embodiment, the first dual-write component may be a dual-write capable group suitable for the second database. Pieces. The first dual-write component can act as an upper-level agent of the second database to receive data transmission and data access requirements. The advantage of this setting is that data transmission and data access with the second database can be realized through the first dual-write component, which can avoid directly affecting the second database when there are problems with data transmission and data access. In the case that there are multiple candidate databases, the first dual-write component can also serve as an upper-level agent for multiple candidate databases, simplifying data transmission and data access procedures. It can be understood that the candidate database includes a second database.

In an embodiment, the second database may be a Redis database, and the first dual-write component is twemproxy.

In an embodiment, the first synchronization condition can be set according to user requirements, which is not limited here. In an embodiment, the first synchronization condition may include, but is not limited to, at least one of the following: the first switching instruction is received; the current data synchronization time reaches the preset first synchronization duration threshold; the first database real time has been received All the historical data of the first database is transmitted, that is, the historical data of the first database has been completely synchronized to the first dual-write component; the to-be-synchronized data to be received in the second database meets a preset condition, where the preset condition includes The amount of received data to be synchronized is within a preset threshold range, etc.

S230: When the second synchronization condition is met, control the second database to write the second incremental data in the second database into the slave database through the first double-write component.

Similarly, the second synchronization condition can be set according to user requirements, which is not limited here. In an embodiment, the second synchronization condition may be the same as the first synchronization condition, or may be different from the first synchronization condition. For example, the first synchronization condition includes that the current data synchronization time reaches a preset first synchronization duration threshold, and the second synchronization condition includes that the current data synchronization time reaches a preset second synchronization duration threshold, and the like.

S240: Control the second database to receive the third incremental data of the slave database written by the slave database through the second dual-write component of the slave database.

Similarly, the second dual-write component may be a component with dual-write capability applicable to the slave database. The second dual-write component can act as the upper-level agent of the slave database to receive data transmission and data access requirements. The advantage of this setting is that data transmission and data access with the slave database can be realized through the second double-write component, which can avoid directly affecting the slave database when there are problems with data transmission and data access. In the case that there are multiple candidate databases, the second dual-write component can also serve as an upper-level agent for multiple candidate databases, simplifying data transmission and data access processes. It can be understood that the candidate database includes the secondary database.

In an embodiment, the slave database may be a Redis database, and the second double write component may be twemproxy. In an embodiment, the third incremental data includes data written into the slave database through the second double-write component of the slave database. Controlling the second database to receive the third incremental data of the secondary database can synchronize the third incremental data of the secondary database to the second database when the third incremental data exists in the secondary database, so that The second incremental data included in the second database is consistent with the data in the slave database.

In an embodiment, if the first database and the slave database receive incremental data at the same time, or the data switching from the first database to the slave database has been completed, that is, when the user data is all supported by the slave database, then the slave database The third incremental data includes incremental data originally input into the first database.

In the technical solution of this embodiment, the first dual-write component set in the second database can provide the second database with dual-write capability, and the method of directly writing data through the first dual-write component and the second dual-write component is suitable for When the historical data in the first database and the second database have been synchronized, only the subsequent incremental data needs to be synchronized; moreover, the first database and the second database are realized through the first data pipeline TPIPE component and the second data pipeline TPIPE component The data transmission can not only realize the orderly transmission of data in batches, but also ensure the real-time performance of data intercommunication, save the time of data synchronization, and improve the efficiency of data synchronization.

In terms of the above technical solutions, in order to truly realize that the user is unaware of data switching and does not affect the normal use of the user at all, the data intercommunication method of the embodiment of the present disclosure may further include: receiving the user's access request from the database In the case of determining the access authority of the user according to the attribute information of the user. In an embodiment, the user's attribute information may include the user's name, gender, age, position, ID number, login account information, and so on.

In an embodiment, determining the user's access authority according to the user's attribute information may include: pre-establishing a user whitelist according to the user's attribute information, and if the user's to-be-matched information corresponding to the access request is If the information in the user whitelist matches, the user is allowed to access the slave database. In an embodiment, the whitelisted user may be a user who manages or views the database, such as a database tester, a database maintainer, and so on.

In order to ensure normal access of ordinary users, if the to-be-matched information of the user corresponding to the access request matches the information in the user whitelist, the user may be regarded as an ordinary user, and the access request may be switched to A database access request. The advantage of this setting is that it can ensure that during the database synchronization process, the staff can access the new library and the ordinary users access the old library. In this way, even if the data needs to be maintained during the data synchronization process, it is ensured that the user has no perception during the test process, does not affect the online, and does not affect the user's access, thereby improving the user experience.

In an embodiment, when the slave database satisfies the conditions for receiving all user accesses, the access requests of all users can be switched to the slave database, so that users who access the first database can simultaneously access the first database and the second database. Database data to realize real-time data intercommunication. Example three

FIG. 3 is a schematic flow chart of a data intercommunication method provided by Embodiment 3 of the present disclosure. This embodiment is suitable for data intercommunication between two independent databases and can be used to realize multiple activities in different places. The method can be implemented by a data intercommunication device. To execute, the device may be configured in a terminal or a server to execute the data intercommunication method of this embodiment.

As shown in FIG. 3, the method of this embodiment may include the following steps:

S310. Control the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component.

In this embodiment, the data to be synchronized includes historical data in the first database and first incremental data in the first database. As mentioned earlier, historical data and first incremental data can be divided according to actual needs. For example, historical data may be data stored at the current moment in the first database; and the first incremental data may be data written after the current moment. The current time may be the time when the first database starts to transmit the data to be synchronized in real time through the first data pipeline TPIPE component.

In one implementation of the embodiment of the present disclosure, the controlling the first database to transmit the data to be synchronized of the first database to the first data pipeline TPIPE component may include: controlling the first database to the first data pipeline TPIPE component Transmit the historical data of the first database in real time; in the case that the first database receives the first incremental data written by the third double-write component to the first database, control the first database to send The first data pipeline TPIPE component transmits the first incremental data in real time. The historical data and the first incremental data can be synchronized in real time to the first data pipeline TPIPE component, so as to import all the data to be synchronized in the first database through the first data pipeline TPIPE component into the second database.

In an embodiment, in order to ensure the security of data intercommunication, before controlling the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component, the method further includes: Data conflict detection is performed on the data of the database. In an embodiment, if there is no data conflict between the data of the first database and the data of the second database, the first database may be controlled to transmit the data to be synchronized of the first database to the first data pipeline TPIPE component. In addition, if there is a data conflict between the data in the first database and the data in the second database, a data conflict prompt message can be generated to remind the user to handle the data conflict.

S320: Control the TPIPE component of the first data pipeline to transmit the to-be-synchronized data of the first database to the second database in real time.

In this embodiment, the first data pipe TPIPE component is used to orderly transmit the data to be synchronized. In this embodiment, the first data pipe TPIPE component complies with the communication protocol of the first database and the second database. In the embodiment of the present disclosure, the first data pipeline TPIPE component receives the data to be synchronized in the first database, and then transmits the data to be synchronized in the first database to the second database in real time, so that the data in the first database The data to be synchronized is imported into the second database in a pipe manner, so that one-way synchronization from the first database to the second database is realized.

In the technical solution of this embodiment, the data to be synchronized in the first database is transmitted to the first data pipe TPIPE component in an orderly manner, and the data to be synchronized is transmitted to the second database by the first data pipe TPIPE component in an orderly manner. That is, the historical data and the first incremental data of the first database are accurately and completely synchronized to the second database through the first data pipeline TPIPE component, so that the second data includes both the data to be synchronized in the first data and Including the local write data of the second database, which realizes data synchronization, solves the technical problems such as error-proneness and poor user experience when the database directly transmits data. It can be in a state of non-stop without affecting the transmission of historical data. It does not affect incremental data and lays a solid foundation for rapid and accurate data exchange.

Example four

FIG. 4 is a schematic flowchart of a data intercommunication method provided by Embodiment 4 of the disclosure. This embodiment is described on the basis of the foregoing embodiment. In order to fully ensure the feasibility of data intercommunication, before starting data transmission, it is also possible to detect whether there is a data conflict between the data of the intercommunication database. In an embodiment, controlling the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component may include: in the case that there is no data conflict between the data in the first database and the second database, Control the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component.

In the embodiment of the present disclosure, controlling the first data pipeline TPIPE component to transmit the data to be synchronized of the first database to the second database in real time includes: controlling the first data pipeline when a third synchronization condition is satisfied The TPIPE component transmits the to-be-synchronized data of the first database to the first double write component of the second database in real time, and controls the first double write component of the second database to write the first database to the second database in real time Transfer data to be synchronized.

As shown in FIG. 4, the method of this embodiment may include the following steps:

S410: When there is no data conflict between the data in the first database and the second database, control the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component.

In an embodiment, the data conflict detection may be performed through the log files of the first database and the second database that record the full amount of data. Taking the Redis database as an example, whether there is a data conflict between the first database and the second database can be detected according to the full file (Append Only File, AOF) backed up by the first database and the second database. Among them, A0F is equivalent to a log of recording operations, and each data change will be additionally recorded in the A0F log. When the service starts, it will read these operation logs and re Perform the recorded operation once to restore the original data. Therefore, data conflicts can be detected by combining the data in the first database AOF and the second database AOF.

S420: When the third synchronization condition is met, control the first data pipeline TPIPE component to transmit the to-be-synchronized data of the first database to the first double-write component of the second database in real time, and control the data of the second database The first dual-write component writes the data to be synchronized that is transmitted in real time by the first database to the second database.

In this embodiment, the third synchronization condition may be the same as the first synchronization condition or the second condition, or may be different from the first synchronization condition and the second synchronization condition.

In an embodiment, the third synchronization condition may also include, but is not limited to, at least one of the following: the first switching instruction is received; the current data synchronization time reaches the preset first synchronization duration threshold; the first synchronization time threshold has been received A database transmits all the historical data of the first database in real time, that is, the historical data of the first database has been completely synchronized to the second data; the data to be synchronized to be received by the second database meets preset conditions, where the preset conditions include waiting The amount of received data to be synchronized is within a preset threshold range, etc.

The technical solution of this embodiment can perform data detection on the first database and the second database before the first database and the second database are interoperable, so as to eliminate the impact of data conflicts on data interoperability, and provide a basis for the feasibility of data interoperability Moreover, when the third synchronization condition is met, the first database is controlled to write to the second database the data to be synchronized transmitted in real time by the first database through the first double write component of the superior agent of the second database, which further ensures The security of data intercommunication.

Example five

FIG. 5a is a data interworking system further provided by an embodiment of the present disclosure. The data interworking system includes: a first database 510, a second database 520, and a secondary database 530 of the second database 520. The first database 510 transmits the data to be synchronized of the first database 510 to the first data pipe TPIPE component, and the first data pipe TPIPE component transmits the data to be synchronized of the first database 510 to the second database 520 in real time; the second database 520 receives the to-be-synchronized data transmitted in real time by the first database 510 through the first data pipeline TPIPE component, and transmits all the data in the second database 520 to the secondary database 530 of the second database 520 through the second data pipeline TPIPE component.

In this embodiment, the data to be synchronized includes the historical data of the first database and the first incremental data in the first database, and the full amount of data includes the data to be synchronized and all the data transmitted by the first database. The local write data in the second database, the first data pipeline TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipeline TPIPE component is used to orderly transmit the full amount of data.

As an optional example of the data intercommunication system in the embodiment of the present disclosure, the first database, the second As an example, the data exchange between the database and the secondary database of the second database is the Redis database, as shown in Figures 5b-5i, the implementation is as follows:

Referring to Figure 5b, the first database T Redis and the second database M Redis are two relatively independent databases, and the first database T Redis and the second database M Redis are respectively set with their respective upper-layer agents twemproxy.

Referring to FIG. 5c, the first database T Redis transmits the to-be-synchronized data of the first database T Redis to the first data pipeline TPIPE component, and the first data pipeline TPIPE component transmits the first database T Redis to the second database in real time. The second database M Redis receives the data to be synchronized transmitted in real time by the first database T Redis through the first data pipeline TPIPE component, and establishes a one-way synchronization from the first database T Redis to the second database M Redis.

Referring to Figure 5d, it should be noted that, since the second database M Redis includes the data to be synchronized transmitted by the first database T Redis, the second database M Redis is represented by the second database M+T Redis at this time. The local end of the database T Redis establishes the slave database M+T Redis of the second database M+T Redis, and then the second database M+T Redis sends the slave database M of the second database M+T Redis through the second data pipeline TPIPE component +T Redis transmits the full amount of data in the second database M+T Redis to achieve two-way data synchronization.

Referring to Figure 5e, when the first switching instruction is received, the first database T Redis is established through the first data pipeline TPIPE component with the first dual-write component twemproxy of the second database M+T Redis to establish a communication connection, so that the first data pipeline The TPIPE component writes the received data to be synchronized in the first database T Redis into the second database M+T Redis through the first dual-write component twemproxy, and the second database M+T Redis receives the waiting data written by the first dual-write component twemproxy. Synchronous Data.

Referring to FIG. 5f, when the second switching instruction is received, the second database M+T Redis writes the second database M+T to the slave database M+T Redis through the first double write component twemproxy of the second database M+T Redis The second incremental data in Redis, that is, when the second incremental data is generated, it can be directly written into the second database M+T Redis and from the database M+T Redis through the first dual write component twemproxy of the second database M+T Redis T Redis.

Referring to Figure 5g, the second incremental data from the database M+T Redis will be written from the database M+T Redis to the second database M+T Redis through the second double write component twemproxy from the database M+T Redis, the second database M+T Redis receives the second incremental data sent from the database M+T Redis.

Referring to Figure 5h, considering the data security issues and user experience issues in the synchronization process, users can be diverted, and a user whitelist can be pre-established according to the user's attribute information; further, when the user's access request is received, it belongs to the whitelist The user accesses from the database M+T Redis, among which whitelist users can be users who have the ability to manage or view the database, such as database testers and database Maintenance personnel, etc., ordinary users who are not in the whitelist temporarily access the local data of the first database T Redis;

Referring to Figure 5i, when the slave database meets the conditions for receiving all user accesses, the access requests of all users can be switched to the slave database M+T Redis, so that users who access the first database T Redis can simultaneously access the first database. The data of the database T Redis and the second database M+T Redis are used to realize real-time data intercommunication.

The technical solution of this embodiment can realize data intercommunication without stopping the machine, and has no influence on the historical data and incremental data in multiple databases, and can also ensure that the test process does not affect the online, thus realizing no Perceived data intercommunication greatly improves user experience.

Example Six

6 is a schematic structural diagram of a data interworking device provided by an embodiment of the disclosure. As shown in FIG. 6, the data interworking device of this embodiment includes a data receiving module 610 and a full data transmission module 620. The data receiving module 610 is configured to receive, through the second database, the data to be synchronized transmitted in real time by the first database through the first data pipeline TPIPE component; the full data transmission module 620 is configured to control the second database through the second data pipeline The TPIPE component transmits the full amount of data in the second database to the slave database of the second database. In this embodiment, the data to be synchronized includes the historical data of the first database and the first incremental data in the first database, and the full amount of data includes the data to be synchronized transmitted by the first database and the The local end writes data in the second database, the first data pipe TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipe TPIPE component is used to orderly transmit the full amount of data.

On the basis of the foregoing embodiment, the data receiving module 610 receives the data to be synchronized that is transmitted in real time by the first database through the first data pipeline TPIPE component through the second database in the following manner: When the first synchronization condition is met, the The first dual-write component of the second database receives the first database communication The data to be synchronized transmitted in real time via the first data pipeline TPIPE component is controlled to receive the data to be synchronized in real time transmitted by the first database written by the first double-write component of the second database by the second database.

On the basis of the foregoing embodiment, the first synchronization condition may include but is not limited to at least one of the following conditions: the first switching instruction is received; the current data synchronization time reaches the preset first synchronization duration threshold; Real-time transmission of all historical data of the first database to the first database; the data to be synchronized to be received in the second database satisfies a preset condition, where the preset condition includes the value of the data to be synchronized to be received The amount of data is within the preset threshold range.

On the basis of the foregoing embodiment, the data intercommunication device may further include: a first writing module, configured to control the second database to write data to the second database through the first dual writing component when the second synchronization condition is satisfied The second incremental data written in the second database is written from the database.

On the basis of the foregoing embodiment, the data intercommunication device may further include: a second writing module configured to control the second database to receive the data written by the slave database through the second dual-write component of the slave database The third incremental data of the slave database.

On the basis of the foregoing embodiment, the data intercommunication device may further include: an authority management module configured to determine the user's access request based on the user's attribute information in the case that the user's access request is received from the database access permission.

The above-mentioned data intercommunication device can execute the data intercommunication method provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method.

Example Seven

FIG. 7 is a schematic structural diagram of a data intercommunication device provided in Embodiment 7 of the present disclosure. As shown in FIG. 7, the data intercommunication device includes: a first data transmission module 710 and a second data transmission module 720. Wherein, the first data transmission module 710 is configured to control the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component; the second data transmission module 720 is configured to control the first data pipeline TPIPE The component transmits the to-be-synchronized data of the first database to the second database in real time; wherein, the to-be-synchronized data includes historical data of the first database and first incremental data in the first database. The data pipeline TPIPE component is used to orderly transmit the data to be synchronized.

In the technical solution of this embodiment, the data to be synchronized in the first database is transmitted to the first data pipe TPIPE component in an orderly manner, and the data to be synchronized is transmitted to the second database by the first data pipe TPIPE component in an orderly manner. That is, the historical data and the first incremental data of the first main database are accurately and completely synchronized to the second database through the first data pipeline TPIPE component, so that the second database includes the data to be synchronized in the first database. It also includes the local write data of the second database, which realizes data synchronization and solves the technical problems such as error-proneness and poor user experience when the database directly transmits data. It can be in a non-stop state without affecting the transmission of historical data. Does not affect incremental data, for The rapid and accurate realization of data interchange lays a solid foundation.

On the basis of the foregoing embodiment, the first data transmission module 710 is configured to: control the first database to transmit the historical data of the first database to the first data pipeline TPIPE component in real time; and receive the historical data in the first database When the third dual-write component writes the first incremental data of the first database, control the first database to transmit the first incremental data to the first data pipeline TPIPE component in real time.

On the basis of the foregoing embodiment, the second data transmission module 720 is configured to: in a case where the third synchronization condition is satisfied, control the first database to write to the second database through the first dual-write component of the second database The second database writes the to-be-synchronized data transmitted in real time from the first database.

On the basis of the foregoing embodiment, the first data transmission module 710 is configured to: control the first database to pass through the first data pipeline when there is no data conflict between the data in the first database and the second database The TPIPE component transmits the data to be synchronized to the second database in real time.

The above-mentioned data intercommunication device can perform the data intercommunication method provided by any embodiment of the present disclosure.

Example eight

Next, referring to FIG. 8, FIG. 8 shows a schematic structural diagram of an electronic device (such as a terminal device or a server) 800 suitable for implementing the embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, PDA), tablets (Portable Android Device, PAD), portable multimedia players ( Portable Media Player (PMP) mobile terminals such as vehicle-mounted terminals (for example, vehicle navigation terminals) and fixed terminals such as digital television (television, TV) desktop computers and the like. The electronic device shown in FIG. 8 is only an example, and should not bring any limitation to the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 8, the electronic device 800 may include a processing device (such as a central processing unit, a graphics processor, etc.) 801, and FIG. 8 may be based on a program stored in a read-only memory (Read-Only Memory, ROM) 802 or from a storage The device 808 loads a program in a random access memory (Random Access Memory, RAM) 803 to perform various appropriate actions and processes. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.

Generally, the following devices can be connected to the I/O interface 805: including input devices 806 such as touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, liquid crystal display (LCD) The output device 807 of a speaker, a vibrator, etc.; a storage device 808 such as a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to perform wireless or wired communication with other devices to exchange data. Although Figure 8 shows The electronic device 800 has multiple devices, but it should be understood that it is not required to implement or have all the illustrated devices. It may alternatively be implemented or provided with more or fewer devices.

According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product. The computer program product includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, it executes the above-mentioned functions defined in the method of the embodiment of the present disclosure.

Example 9

The aforementioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, RAM, ROM, Erasable Programmable Read-Only Memory, EPROM or flash memory), optical fiber, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium may include a data signal propagated in the baseband or as a part of a carrier wave, and the computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium. The computer-readable signal medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device . The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist alone without being assembled into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: receives the first database through the second database and transmits it in real time through the first data pipeline TPIPE component Data to be synchronized; controlling the second database to transmit data in the second database to the second database through the second data pipeline TPIPE component Full data; wherein, the data to be synchronized includes historical data of the first database and first incremental data in the first database, and the full data includes the data to be synchronized and all data transmitted by the first database The local write data in the second database, the first data pipeline TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipeline TPIPE component is used to orderly transmit the full amount of data.

Alternatively, the aforementioned computer-readable medium carries one or more programs, and when the aforementioned one or more programs are executed by the electronic device, the electronic device is caused to: control the first database to transmit the first data pipeline TPIPE component. The data to be synchronized in the database; controlling the first data pipeline TPIPE component to transmit the data to be synchronized in the first database in real time to the second database; wherein the data to be synchronized includes the historical data of the first database and the The first incremental data in the first database, the first data pipeline TPIPE component is used to orderly transmit the data to be synchronized.

The computer program code used to perform the operations of the present disclosure can be written in one or more programming languages or a combination thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also conventional Procedural programming language-such as "C" language or similar programming language. The program code may be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server. In the case of a remote computer, the remote computer can pass through any kind of network-including a local area network ((Local Area Network,

LAN) or Wide Area Network (Wide Area Network, WAN)—Connect to a user's computer, or, alternatively, it can be connected to an external computer (for example, using an Internet service provider to connect via the Internet).

The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation of the system architecture, functions, and operations of the method and computer program product according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the module, program segment, or part of code includes one or more for realizing prescribed logical functions Executable instructions. It should also be noted that, in some alternative implementations, the functions marked in the blocks may also occur in a different order from the order marked in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation Or it can be implemented by a combination of dedicated hardware and computer instructions.

The modules and units involved in the described embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner. Among them, the name of the module or unit does not constitute a limitation on the module or unit itself under certain circumstances.

Claims

1. A data intercommunication method, including:

Controlling the second database to transmit the full amount of data in the second database to the secondary database of the second database through the second data pipeline TPIPE component;

Wherein, the data to be synchronized includes historical data of the first database and first incremental data in the first database, and the full amount of data includes the data to be synchronized transmitted by the first database and the second The local end writes data in the database, the first data pipe TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipe TPIPE component is used to orderly transmit the full amount of data.

2. The method according to claim 1, wherein the receiving, through the second database, the data to be synchronized which is transmitted in real time by the first database through the TPIPE component of the first data pipeline comprises:

In the case that the first synchronization condition is met, the first dual-write component of the second database receives the data to be synchronized that the first database transmits in real time through the first data pipeline TPIPE component, and controls the second database to receive the second database The data to be synchronized for real-time transmission of the first database written by the first dual-write component.

3. The method according to claim 2, wherein the first synchronization condition includes at least one of the following conditions:

Receiving the first switching instruction;

The current data synchronization time reaches the preset first synchronization duration threshold;

The first database has been received to transmit all the historical data of the first database in real time; the to-be-synchronized data to be received by the second database satisfies a preset condition, where the preset condition includes the to-be-synchronized to be received The amount of data is within a preset threshold range.

4. The method according to claim 2 or 3, further comprising:

When the second synchronization condition is satisfied, the second database is controlled to write the second incremental data in the second database into the secondary database of the second database through the first dual-write component.

5. The method according to claim 4, further comprising:

The second database is controlled to receive the third incremental data of the slave database written by the slave database of the second database through the second double-write component of the slave database of the second database.

6. The method according to any one of claims 1-5, further comprising:

In a case where the user's access request is received from the database, the user's access authority is determined according to the attribute information of the user.

7. A data intercommunication method, including:

8. The method according to claim 7, wherein the controlling the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component comprises:

Controlling the first database to transmit the historical data of the first database to the first data pipeline TPIPE component in real time;

When the first database receives the first incremental data written by the dual-write component to the first database, control the first database to transmit the first incremental data to the first data pipeline TPIPE component in real time量数据。 Volume data.

9. The method according to claim 7, wherein the controlling the TPIPE component of the first data pipeline to transmit the data to be synchronized of the first database to the second database in real time comprises:

When the synchronization condition is met, control the first data pipeline TPIPE component to transmit the data to be synchronized in the first database in real time to the first double write component of the second database, and control the first double write of the second database The component writes to the second database the data to be synchronized transmitted in real time by the first database.

10. The method according to claim 7, wherein the controlling the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component comprises:

If there is no data conflict between the data in the first database and the second database, control the first database to transmit the data to be synchronized of the first database to the first data pipeline TPIPE component.

11. A data intercommunication device, including:

Wherein, the data to be synchronized includes the historical data of the first database and the first incremental data in the first database, and the full amount of data includes the data to be synchronized transmitted by the first database and the The local end writes data in the second database, the first data pipe TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipe TPIPE component is used to orderly transmit the full amount of data.

12. A data intercommunication device, including:

13. An electronic device, including:

At least one processing device; a memory, configured to store at least one program;

When the at least one program is executed by the at least one processing device, the at least one processing device realizes the data intercommunication method according to any one of claims 1-9.

14. A computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, realizes the data intercommunication method according to any one of claims 1-9.