WO2020173381A1 - Procédé et dispositif d'interfonctionnement de données, terminal et support de stockage - Google Patents

Procédé et dispositif d'interfonctionnement de données, terminal et support de stockage Download PDF

Info

Publication number
WO2020173381A1
WO2020173381A1 PCT/CN2020/076050 CN2020076050W WO2020173381A1 WO 2020173381 A1 WO2020173381 A1 WO 2020173381A1 CN 2020076050 W CN2020076050 W CN 2020076050W WO 2020173381 A1 WO2020173381 A1 WO 2020173381A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
database
component
synchronized
tpipe
Prior art date
Application number
PCT/CN2020/076050
Other languages
English (en)
Chinese (zh)
Inventor
Weijie SONG (宋维捷)
Original Assignee
北京字节跳动网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京字节跳动网络技术有限公司 filed Critical 北京字节跳动网络技术有限公司
Publication of WO2020173381A1 publication Critical patent/WO2020173381A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Definitions

  • the embodiments of the present disclosure relate to the field of computer application technology, for example, to a data intercommunication method, device, terminal, and storage medium. Background technique
  • Remote Dictionary Server (Redis) is a non-relational memory database that is widely used in Nosql databases. Inside Redis is a key-value storage system. Because Redis has the advantages of easy expansion, large data volume to improve performance, and diverse and flexible data models, it is widely used.
  • Multi-activity in different places generally refers to the establishment of independent data centers in different cities. These computer rooms also need to carry traffic in daily business to provide business support. That is, data exchange is performed between the databases of two computer rooms in different cities, so that users can access data in one place in different cities.
  • the Redis database data intercommunication method of related technologies usually directly transfers the data between the two Redis databases. Because the Redis database may include historical data and write in real time Incremental data makes it extremely prone to errors during data transmission, and it is often difficult to achieve data intercommunication without affecting the data, and the user experience is poor. Summary of the invention
  • the embodiments of the present disclosure provide a data intercommunication method, device, terminal, and storage medium to solve technical problems such as error-proneness and poor user experience when directly transmitting data between a database and a database in related technologies, and can achieve data intercommunication simply and effectively.
  • the embodiment of the present disclosure provides a data intercommunication method, which includes:
  • the data to be synchronized includes historical data of the first database and first incremental data in the first database
  • the full amount of data includes the data to be synchronized transmitted by the first database and the second
  • the local end writes data in the database
  • the first data pipe TPIPE component is used to orderly transmit the data to be synchronized
  • the second data pipe TPIPE component is used to orderly transmit the full amount of data.
  • the embodiment of the present disclosure also provides a data intercommunication method, which includes:
  • the data to be synchronized includes historical data of the first database and first incremental data in the first database, and the first data pipeline TPIPE component is used for orderly transmitting the data to be synchronized.
  • the embodiment of the present disclosure also provides a data intercommunication device, which includes:
  • a data receiving module configured to receive, through the second database, the data to be synchronized that the first database transmits in real time through the TPIPE component of the first data pipeline;
  • a full data transmission module configured to control the second database to transmit the full data in the second database to the second database through the second data pipeline TPIPE component;
  • the data to be synchronized includes historical data of the first database and first incremental data in the first database, and the full amount of data includes the data to be synchronized transmitted by the first database and the second database
  • the local end writes data in, the first data pipe TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipe TPIPE component is used to orderly transmit the full amount of data.
  • the embodiment of the present disclosure also provides a data intercommunication device, which includes:
  • the first data transmission module is configured to control the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component;
  • the second data transmission module is configured to control the first data pipeline TPIPE component to transmit the to-be-synchronized data of the first database to the second database in real time;
  • the data to be synchronized includes historical data of the first database and first incremental data in the first database.
  • the first data pipeline TPIPE component is used to orderly transmit the data to be synchronized.
  • the embodiments of the present disclosure also provide an electronic device, which includes:
  • One or more processing devices are One or more processing devices;
  • Memory set to store one or more programs; When the one or more programs are executed by the one or more processing apparatuses, the one or more processing apparatuses implement the data intercommunication method according to any embodiment of the present disclosure.
  • the embodiments of the present disclosure also provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the data intercommunication method as described in any embodiment of the present disclosure is implemented. Description of the drawings
  • Fig. 1 is a schematic flowchart of a data interworking method provided in the first embodiment of the present disclosure
  • Fig. 2 is a schematic flowchart of a data interworking method provided in the second embodiment of the present disclosure
  • Fig. 4 is a schematic flowchart of a data intercommunication method provided by Embodiment 4 of the present disclosure
  • Fig. 5a is a schematic flowchart of a data intercommunication system provided by Embodiment 5 of the present disclosure
  • 5b is a schematic diagram of one-way data synchronization provided in the fifth embodiment of the present disclosure
  • Figure 5c is a schematic diagram of the interaction of two-way data synchronization provided in the fifth embodiment of the present disclosure
  • Figure 5d is another schematic diagram of the interaction of two-way data synchronization provided in the fifth embodiment of the present disclosure
  • Figure 5e is another schematic diagram of the two-way data synchronization provided in the fifth embodiment of the present disclosure
  • Figure 5f is another schematic diagram of the interaction of two-way data synchronization provided by the fifth embodiment of the present disclosure
  • Figure 5g is another schematic diagram of the interaction of two-way data synchronization provided by the fifth embodiment of the present disclosure
  • 5h is another schematic diagram of the interaction of two-way data synchronization provided by the fifth embodiment of the present disclosure
  • FIG. 5i is a schematic diagram of the data interaction for data intercommunication provided by the fifth embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of the data interaction provided by the sixth embodiment of the present disclosure
  • Fig. 7 is a schematic structural diagram of a data intercommunication device provided in the seventh embodiment of the present disclosure
  • Fig. 8 is a schematic structural diagram of an electronic device provided in the eighth embodiment of the present disclosure.
  • FIG. 1 is a schematic flowchart of a data intercommunication method provided in Embodiment 1 of the present disclosure.
  • This example is suitable for data intercommunication between two independent databases, which can be used to realize multiple activities in different places.
  • This method can be executed by a data intercommunication device, and the device can be configured in a terminal or a server to execute the data intercommunication method of this embodiment.
  • whether the two databases are in different places can be determined according to the round-trip time (Round-Trip Time, RTT) between the two databases. That is, according to the total time delay experienced from the sending end of the data transmission until the sending end receives the confirmation from the receiving end, the receiving end generally sends the confirmation immediately after receiving the data.
  • RTT Round-trip Time
  • the method of this embodiment may include the following steps:
  • S110 Receive, through the second database, the data to be synchronized that is transmitted in real time by the first database through the TPIPE component of the first data pipeline.
  • the data to be synchronized includes historical data in the first database and first incremental data in the first database.
  • the first data pipeline TPIPE component is used to orderly transmit the to-be-synchronized data of the first database.
  • the historical data and the first incremental data can be divided according to actual needs.
  • the historical data may be data stored at the current time in the first database; the first incremental data may be data written after the current time.
  • the current moment may be the moment when the first database starts to transmit the data to be synchronized in real time through the first data pipeline TPIPE component.
  • a large amount of to-be-synchronized data transmitted in real time by the first database through the TPIPE component of the first data pipeline is received through the second database, that is, one-way synchronization between the second database and the first database is established through the TPIPE component of the first data pipeline.
  • the first database and the second database may include non-relational memory databases.
  • the first database and the second database may both be Redis databases.
  • the full amount of data includes the data to be synchronized transmitted by the first database and the local write data in the second database.
  • the second database has received the data to be synchronized from the first database.
  • the second database includes both the to-be-synchronized data transmitted by the first database and the locally written data in the second database.
  • the first data pipe TPIPE component and the second data pipe TPIPE component may be message queues for orderly transmission of data.
  • the second data pipe TPIPE component can be used to orderly transmit the full amount of data.
  • the method before the second database transmits the full amount of data to the secondary database of the second database through the second data pipeline TPIPE component, the method includes: establishing the secondary database of the second database.
  • the slave database can replace the data access of the first database, and provide data in the first database and the second database at the same time, so as to realize the The data of the first database and the second database are interconnected.
  • the first database may also be called the first main database
  • the second database may also be called the second main database
  • the second database includes the data written by the local end and the data to be synchronized transmitted by the first database received through the first data pipeline TPIPE component, so that users of the second database can access the first database at the same time And the data of the second database; and then transfer the full amount of data in the second database to the secondary database of the second database, so that users who originally accessed the first database can simultaneously access the first database and the second database through the secondary database of the second database Database data realizes data intercommunication.
  • the orderly transmission of data can be realized through the first data pipeline TPIPE component and the second data pipeline TPIPE component, which can ensure the integrity and real-time performance of data transmission, and solve the removal of data intercommunication methods.
  • the technical problems of storage and poor user experience can neither affect the transmission of historical data nor the transmission of incremental data without stopping the machine, and truly realize the data intercommunication of the database.
  • FIG. 2 is a schematic flowchart of a data intercommunication method provided in Embodiment 2 of the present disclosure. This embodiment is described on the basis of the foregoing embodiment.
  • receiving, through the second database, the data to be synchronized that is transmitted in real time by the first database through the first data pipeline TPIPE component includes: receiving through the first dual-write component of the second database when the first synchronization condition is satisfied
  • the first database transmits the data to be synchronized in real time through the first data pipeline TPIPE component, and controls the second database to receive the data to be synchronized in real time transmitted by the first database written by the first dual-write component of the second database.
  • the data intercommunication method of this embodiment may further include: in a case where a second synchronization condition is met, controlling the second database to write the slave database to the slave database through the first dual-write component The second incremental data in the second database.
  • the data intercommunication method of this embodiment may further include: controlling the second database to receive the third increment of the slave database written by the slave database through the second double-write component of the slave database data.
  • the method of this embodiment may include the following steps:
  • the first dual-write component may be a dual-write capable group suitable for the second database. Pieces.
  • the first dual-write component can act as an upper-level agent of the second database to receive data transmission and data access requirements.
  • the advantage of this setting is that data transmission and data access with the second database can be realized through the first dual-write component, which can avoid directly affecting the second database when there are problems with data transmission and data access.
  • the first dual-write component can also serve as an upper-level agent for multiple candidate databases, simplifying data transmission and data access procedures. It can be understood that the candidate database includes a second database.
  • the second database may be a Redis database
  • the first dual-write component is twemproxy
  • the first synchronization condition can be set according to user requirements, which is not limited here.
  • the first synchronization condition may include, but is not limited to, at least one of the following: the first switching instruction is received; the current data synchronization time reaches the preset first synchronization duration threshold; the first database real time has been received All the historical data of the first database is transmitted, that is, the historical data of the first database has been completely synchronized to the first dual-write component; the to-be-synchronized data to be received in the second database meets a preset condition, where the preset condition includes The amount of received data to be synchronized is within a preset threshold range, etc.
  • the second synchronization condition can be set according to user requirements, which is not limited here.
  • the second synchronization condition may be the same as the first synchronization condition, or may be different from the first synchronization condition.
  • the first synchronization condition includes that the current data synchronization time reaches a preset first synchronization duration threshold
  • the second synchronization condition includes that the current data synchronization time reaches a preset second synchronization duration threshold, and the like.
  • S240 Control the second database to receive the third incremental data of the slave database written by the slave database through the second dual-write component of the slave database.
  • the second dual-write component may be a component with dual-write capability applicable to the slave database.
  • the second dual-write component can act as the upper-level agent of the slave database to receive data transmission and data access requirements.
  • the advantage of this setting is that data transmission and data access with the slave database can be realized through the second double-write component, which can avoid directly affecting the slave database when there are problems with data transmission and data access.
  • the second dual-write component can also serve as an upper-level agent for multiple candidate databases, simplifying data transmission and data access processes. It can be understood that the candidate database includes the secondary database.
  • the slave database may be a Redis database
  • the second double write component may be twemproxy
  • the third incremental data includes data written into the slave database through the second double-write component of the slave database. Controlling the second database to receive the third incremental data of the secondary database can synchronize the third incremental data of the secondary database to the second database when the third incremental data exists in the secondary database, so that The second incremental data included in the second database is consistent with the data in the slave database.
  • the slave database receives incremental data at the same time, or the data switching from the first database to the slave database has been completed, that is, when the user data is all supported by the slave database, then the slave database
  • the third incremental data includes incremental data originally input into the first database.
  • the first dual-write component set in the second database can provide the second database with dual-write capability, and the method of directly writing data through the first dual-write component and the second dual-write component is suitable for
  • the historical data in the first database and the second database have been synchronized, only the subsequent incremental data needs to be synchronized; moreover, the first database and the second database are realized through the first data pipeline TPIPE component and the second data pipeline TPIPE component
  • the data transmission can not only realize the orderly transmission of data in batches, but also ensure the real-time performance of data intercommunication, save the time of data synchronization, and improve the efficiency of data synchronization.
  • the data intercommunication method of the embodiment of the present disclosure may further include: receiving the user's access request from the database In the case of determining the access authority of the user according to the attribute information of the user.
  • the user's attribute information may include the user's name, gender, age, position, ID number, login account information, and so on.
  • determining the user's access authority according to the user's attribute information may include: pre-establishing a user whitelist according to the user's attribute information, and if the user's to-be-matched information corresponding to the access request is If the information in the user whitelist matches, the user is allowed to access the slave database.
  • the whitelisted user may be a user who manages or views the database, such as a database tester, a database maintainer, and so on.
  • the user may be regarded as an ordinary user, and the access request may be switched to A database access request.
  • the advantage of this setting is that it can ensure that during the database synchronization process, the staff can access the new library and the ordinary users access the old library. In this way, even if the data needs to be maintained during the data synchronization process, it is ensured that the user has no perception during the test process, does not affect the online, and does not affect the user's access, thereby improving the user experience.
  • the access requests of all users can be switched to the slave database, so that users who access the first database can simultaneously access the first database and the second database.
  • Database data to realize real-time data intercommunication Example three
  • FIG. 3 is a schematic flow chart of a data intercommunication method provided by Embodiment 3 of the present disclosure.
  • This embodiment is suitable for data intercommunication between two independent databases and can be used to realize multiple activities in different places.
  • the method can be implemented by a data intercommunication device.
  • the device may be configured in a terminal or a server to execute the data intercommunication method of this embodiment.
  • the method of this embodiment may include the following steps:
  • the data to be synchronized includes historical data in the first database and first incremental data in the first database.
  • historical data and first incremental data can be divided according to actual needs.
  • historical data may be data stored at the current moment in the first database; and the first incremental data may be data written after the current moment.
  • the current time may be the time when the first database starts to transmit the data to be synchronized in real time through the first data pipeline TPIPE component.
  • controlling the first database to transmit the data to be synchronized of the first database to the first data pipeline TPIPE component may include: controlling the first database to the first data pipeline TPIPE component Transmit the historical data of the first database in real time; in the case that the first database receives the first incremental data written by the third double-write component to the first database, control the first database to send The first data pipeline TPIPE component transmits the first incremental data in real time.
  • the historical data and the first incremental data can be synchronized in real time to the first data pipeline TPIPE component, so as to import all the data to be synchronized in the first database through the first data pipeline TPIPE component into the second database.
  • the method further includes: Data conflict detection is performed on the data of the database.
  • the first database may be controlled to transmit the data to be synchronized of the first database to the first data pipeline TPIPE component.
  • a data conflict prompt message can be generated to remind the user to handle the data conflict.
  • S320 Control the TPIPE component of the first data pipeline to transmit the to-be-synchronized data of the first database to the second database in real time.
  • the first data pipe TPIPE component is used to orderly transmit the data to be synchronized.
  • the first data pipe TPIPE component complies with the communication protocol of the first database and the second database.
  • the first data pipeline TPIPE component receives the data to be synchronized in the first database, and then transmits the data to be synchronized in the first database to the second database in real time, so that the data in the first database
  • the data to be synchronized is imported into the second database in a pipe manner, so that one-way synchronization from the first database to the second database is realized.
  • the data to be synchronized in the first database is transmitted to the first data pipe TPIPE component in an orderly manner, and the data to be synchronized is transmitted to the second database by the first data pipe TPIPE component in an orderly manner. That is, the historical data and the first incremental data of the first database are accurately and completely synchronized to the second database through the first data pipeline TPIPE component, so that the second data includes both the data to be synchronized in the first data and Including the local write data of the second database, which realizes data synchronization, solves the technical problems such as error-proneness and poor user experience when the database directly transmits data. It can be in a state of non-stop without affecting the transmission of historical data. It does not affect incremental data and lays a solid foundation for rapid and accurate data exchange.
  • FIG. 4 is a schematic flowchart of a data intercommunication method provided by Embodiment 4 of the disclosure. This embodiment is described on the basis of the foregoing embodiment.
  • controlling the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component may include: in the case that there is no data conflict between the data in the first database and the second database, Control the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component.
  • controlling the first data pipeline TPIPE component to transmit the data to be synchronized of the first database to the second database in real time includes: controlling the first data pipeline when a third synchronization condition is satisfied
  • the TPIPE component transmits the to-be-synchronized data of the first database to the first double write component of the second database in real time, and controls the first double write component of the second database to write the first database to the second database in real time Transfer data to be synchronized.
  • the method of this embodiment may include the following steps:
  • the data conflict detection may be performed through the log files of the first database and the second database that record the full amount of data.
  • the full file Append Only File, AOF
  • A0F is equivalent to a log of recording operations, and each data change will be additionally recorded in the A0F log.
  • the service starts, it will read these operation logs and re Perform the recorded operation once to restore the original data. Therefore, data conflicts can be detected by combining the data in the first database AOF and the second database AOF.
  • the third synchronization condition may be the same as the first synchronization condition or the second condition, or may be different from the first synchronization condition and the second synchronization condition.
  • the third synchronization condition may also include, but is not limited to, at least one of the following: the first switching instruction is received; the current data synchronization time reaches the preset first synchronization duration threshold; the first synchronization time threshold has been received
  • a database transmits all the historical data of the first database in real time, that is, the historical data of the first database has been completely synchronized to the second data; the data to be synchronized to be received by the second database meets preset conditions, where the preset conditions include waiting The amount of received data to be synchronized is within a preset threshold range, etc.
  • the technical solution of this embodiment can perform data detection on the first database and the second database before the first database and the second database are interoperable, so as to eliminate the impact of data conflicts on data interoperability, and provide a basis for the feasibility of data interoperability Moreover, when the third synchronization condition is met, the first database is controlled to write to the second database the data to be synchronized transmitted in real time by the first database through the first double write component of the superior agent of the second database, which further ensures The security of data intercommunication.
  • FIG. 5a is a data interworking system further provided by an embodiment of the present disclosure.
  • the data interworking system includes: a first database 510, a second database 520, and a secondary database 530 of the second database 520.
  • the first database 510 transmits the data to be synchronized of the first database 510 to the first data pipe TPIPE component, and the first data pipe TPIPE component transmits the data to be synchronized of the first database 510 to the second database 520 in real time;
  • the second database 520 receives the to-be-synchronized data transmitted in real time by the first database 510 through the first data pipeline TPIPE component, and transmits all the data in the second database 520 to the secondary database 530 of the second database 520 through the second data pipeline TPIPE component.
  • the data to be synchronized includes the historical data of the first database and the first incremental data in the first database, and the full amount of data includes the data to be synchronized and all the data transmitted by the first database.
  • the local write data in the second database, the first data pipeline TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipeline TPIPE component is used to orderly transmit the full amount of data.
  • the first database As an optional example of the data intercommunication system in the embodiment of the present disclosure, the first database, the second As an example, the data exchange between the database and the secondary database of the second database is the Redis database, as shown in Figures 5b-5i, the implementation is as follows:
  • the first database T Redis and the second database M Redis are two relatively independent databases, and the first database T Redis and the second database M Redis are respectively set with their respective upper-layer agents twemproxy.
  • the first database T Redis transmits the to-be-synchronized data of the first database T Redis to the first data pipeline TPIPE component, and the first data pipeline TPIPE component transmits the first database T Redis to the second database in real time.
  • the second database M Redis receives the data to be synchronized transmitted in real time by the first database T Redis through the first data pipeline TPIPE component, and establishes a one-way synchronization from the first database T Redis to the second database M Redis.
  • the second database M Redis since the second database M Redis includes the data to be synchronized transmitted by the first database T Redis, the second database M Redis is represented by the second database M+T Redis at this time.
  • the local end of the database T Redis establishes the slave database M+T Redis of the second database M+T Redis, and then the second database M+T Redis sends the slave database M of the second database M+T Redis through the second data pipeline TPIPE component +T Redis transmits the full amount of data in the second database M+T Redis to achieve two-way data synchronization.
  • the first database T Redis is established through the first data pipeline TPIPE component with the first dual-write component twemproxy of the second database M+T Redis to establish a communication connection, so that the first data pipeline
  • the TPIPE component writes the received data to be synchronized in the first database T Redis into the second database M+T Redis through the first dual-write component twemproxy, and the second database M+T Redis receives the waiting data written by the first dual-write component twemproxy. Synchronous Data.
  • the second database M+T Redis writes the second database M+T to the slave database M+T Redis through the first double write component twemproxy of the second database M+T Redis
  • the second incremental data in Redis that is, when the second incremental data is generated, it can be directly written into the second database M+T Redis and from the database M+T Redis through the first dual write component twemproxy of the second database M+T Redis T Redis.
  • the second incremental data from the database M+T Redis will be written from the database M+T Redis to the second database M+T Redis through the second double write component twemproxy from the database M+T Redis, the second database M+T Redis receives the second incremental data sent from the database M+T Redis.
  • a user whitelist can be pre-established according to the user's attribute information; further, when the user's access request is received, it belongs to the whitelist
  • the user accesses from the database M+T Redis, among which whitelist users can be users who have the ability to manage or view the database, such as database testers and database Maintenance personnel, etc., ordinary users who are not in the whitelist temporarily access the local data of the first database T Redis;
  • the access requests of all users can be switched to the slave database M+T Redis, so that users who access the first database T Redis can simultaneously access the first database.
  • the data of the database T Redis and the second database M+T Redis are used to realize real-time data intercommunication.
  • the technical solution of this embodiment can realize data intercommunication without stopping the machine, and has no influence on the historical data and incremental data in multiple databases, and can also ensure that the test process does not affect the online, thus realizing no Perceived data intercommunication greatly improves user experience.
  • the data interworking device of this embodiment includes a data receiving module 610 and a full data transmission module 620.
  • the data receiving module 610 is configured to receive, through the second database, the data to be synchronized transmitted in real time by the first database through the first data pipeline TPIPE component;
  • the full data transmission module 620 is configured to control the second database through the second data pipeline
  • the TPIPE component transmits the full amount of data in the second database to the slave database of the second database.
  • the data to be synchronized includes the historical data of the first database and the first incremental data in the first database, and the full amount of data includes the data to be synchronized transmitted by the first database and the The local end writes data in the second database, the first data pipe TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipe TPIPE component is used to orderly transmit the full amount of data.
  • the second database includes the data written by the local end and the data to be synchronized transmitted by the first database received through the first data pipeline TPIPE component, so that users of the second database can access the first database at the same time And the data of the second database; and then transfer the full amount of data in the second database to the secondary database of the second database, so that users who originally accessed the first database can simultaneously access the first database and the second database through the secondary database of the second database Database data realizes data intercommunication.
  • the orderly transmission of data can be realized through the first data pipeline TPIPE component and the second data pipeline TPIPE component, which can ensure the integrity and real-time performance of data transmission, and solve the removal of data intercommunication methods.
  • the technical problems of storage and poor user experience can neither affect the transmission of historical data nor the transmission of incremental data without stopping the machine, and truly realize the data intercommunication of the database.
  • the data receiving module 610 receives the data to be synchronized that is transmitted in real time by the first database through the first data pipeline TPIPE component through the second database in the following manner:
  • the The first dual-write component of the second database receives the first database communication
  • the data to be synchronized transmitted in real time via the first data pipeline TPIPE component is controlled to receive the data to be synchronized in real time transmitted by the first database written by the first double-write component of the second database by the second database.
  • the first synchronization condition may include but is not limited to at least one of the following conditions: the first switching instruction is received; the current data synchronization time reaches the preset first synchronization duration threshold; Real-time transmission of all historical data of the first database to the first database; the data to be synchronized to be received in the second database satisfies a preset condition, where the preset condition includes the value of the data to be synchronized to be received The amount of data is within the preset threshold range.
  • the data intercommunication device may further include: a first writing module, configured to control the second database to write data to the second database through the first dual writing component when the second synchronization condition is satisfied The second incremental data written in the second database is written from the database.
  • a first writing module configured to control the second database to write data to the second database through the first dual writing component when the second synchronization condition is satisfied The second incremental data written in the second database is written from the database.
  • the data intercommunication device may further include: a second writing module configured to control the second database to receive the data written by the slave database through the second dual-write component of the slave database The third incremental data of the slave database.
  • the data intercommunication device may further include: an authority management module configured to determine the user's access request based on the user's attribute information in the case that the user's access request is received from the database access permission.
  • the above-mentioned data intercommunication device can execute the data intercommunication method provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method.
  • FIG. 7 is a schematic structural diagram of a data intercommunication device provided in Embodiment 7 of the present disclosure.
  • the data intercommunication device includes: a first data transmission module 710 and a second data transmission module 720.
  • the first data transmission module 710 is configured to control the first database to transmit the to-be-synchronized data of the first database to the first data pipeline TPIPE component;
  • the second data transmission module 720 is configured to control the first data pipeline TPIPE
  • the component transmits the to-be-synchronized data of the first database to the second database in real time; wherein, the to-be-synchronized data includes historical data of the first database and first incremental data in the first database.
  • the data pipeline TPIPE component is used to orderly transmit the data to be synchronized.
  • the data to be synchronized in the first database is transmitted to the first data pipe TPIPE component in an orderly manner, and the data to be synchronized is transmitted to the second database by the first data pipe TPIPE component in an orderly manner. That is, the historical data and the first incremental data of the first main database are accurately and completely synchronized to the second database through the first data pipeline TPIPE component, so that the second database includes the data to be synchronized in the first database. It also includes the local write data of the second database, which realizes data synchronization and solves the technical problems such as error-proneness and poor user experience when the database directly transmits data. It can be in a non-stop state without affecting the transmission of historical data. Does not affect incremental data, for The rapid and accurate realization of data interchange lays a solid foundation.
  • the first data transmission module 710 is configured to: control the first database to transmit the historical data of the first database to the first data pipeline TPIPE component in real time; and receive the historical data in the first database
  • control the first database to transmit the first incremental data to the first data pipeline TPIPE component in real time.
  • the second data transmission module 720 is configured to: in a case where the third synchronization condition is satisfied, control the first database to write to the second database through the first dual-write component of the second database The second database writes the to-be-synchronized data transmitted in real time from the first database.
  • the first data transmission module 710 is configured to: control the first database to pass through the first data pipeline when there is no data conflict between the data in the first database and the second database
  • the TPIPE component transmits the data to be synchronized to the second database in real time.
  • the above-mentioned data intercommunication device can perform the data intercommunication method provided by any embodiment of the present disclosure.
  • FIG. 8 shows a schematic structural diagram of an electronic device (such as a terminal device or a server) 800 suitable for implementing the embodiments of the present disclosure.
  • the terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, PDA), tablets (Portable Android Device, PAD), portable multimedia players ( Portable Media Player (PMP) mobile terminals such as vehicle-mounted terminals (for example, vehicle navigation terminals) and fixed terminals such as digital television (television, TV) desktop computers and the like.
  • PDA Personal Digital Assistant
  • PDA Personal Digital Assistant
  • PAD Portable Multimedia Player
  • PMP Portable Media Player
  • vehicle-mounted terminals for example, vehicle navigation terminals
  • fixed terminals such as digital television (television, TV) desktop computers and the like.
  • the electronic device shown in FIG. 8 is only an example, and should not bring any limitation to the function and scope of use of the embodiments of the present disclosure.
  • the electronic device 800 may include a processing device (such as a central processing unit, a graphics processor, etc.) 801, and FIG. 8 may be based on a program stored in a read-only memory (Read-Only Memory, ROM) 802 or from a storage
  • the device 808 loads a program in a random access memory (Random Access Memory, RAM) 803 to perform various appropriate actions and processes.
  • RAM Random Access Memory
  • various programs and data required for the operation of the electronic device 800 are also stored.
  • the processing device 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804.
  • An input/output (I/O) interface 805 is also connected to the bus 804.
  • the following devices can be connected to the I/O interface 805: including input devices 806 such as touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, liquid crystal display (LCD)
  • input devices 806 such as touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.
  • LCD liquid crystal display
  • a storage device 808 such as a magnetic tape, a hard disk, etc.
  • the communication device 809 may allow the electronic device 800 to perform wireless or wired communication with other devices to exchange data.
  • Figure 8 shows The electronic device 800 has multiple devices, but it should be understood that it is not required to implement or have all the illustrated devices. It may alternatively be implemented or provided with more or fewer devices.
  • the process described above with reference to the flowchart can be implemented as a computer software program.
  • the embodiments of the present disclosure include a computer program product.
  • the computer program product includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart.
  • the computer program may be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802.
  • the computer program is executed by the processing device 801, it executes the above-mentioned functions defined in the method of the embodiment of the present disclosure.
  • the aforementioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, RAM, ROM, Erasable Programmable Read-Only Memory, EPROM or flash memory), optical fiber, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
  • the computer-readable signal medium may include a data signal propagated in the baseband or as a part of a carrier wave, and the computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
  • the computer-readable signal medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wire, optical cable, radio frequency (RF), etc., or any suitable combination of the foregoing.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist alone without being assembled into the electronic device.
  • the above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: receives the first database through the second database and transmits it in real time through the first data pipeline TPIPE component Data to be synchronized; controlling the second database to transmit data in the second database to the second database through the second data pipeline TPIPE component Full data; wherein, the data to be synchronized includes historical data of the first database and first incremental data in the first database, and the full data includes the data to be synchronized and all data transmitted by the first database
  • the local write data in the second database, the first data pipeline TPIPE component is used to orderly transmit the data to be synchronized, and the second data pipeline TPIPE component is used to orderly transmit the full amount of data.
  • the aforementioned computer-readable medium carries one or more programs, and when the aforementioned one or more programs are executed by the electronic device, the electronic device is caused to: control the first database to transmit the first data pipeline TPIPE component.
  • the data to be synchronized in the database controlling the first data pipeline TPIPE component to transmit the data to be synchronized in the first database in real time to the second database; wherein the data to be synchronized includes the historical data of the first database and the The first incremental data in the first database, the first data pipeline TPIPE component is used to orderly transmit the data to be synchronized.
  • the computer program code used to perform the operations of the present disclosure can be written in one or more programming languages or a combination thereof.
  • the above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also conventional Procedural programming language-such as "C" language or similar programming language.
  • the program code may be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server.
  • the remote computer can pass through any kind of network-including a local area network ((Local Area Network,
  • LAN Local Area Network
  • WAN Wide Area Network
  • each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the module, program segment, or part of code includes one or more for realizing prescribed logical functions Executable instructions.
  • the functions marked in the blocks may also occur in a different order from the order marked in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or operation Or it can be implemented by a combination of dedicated hardware and computer instructions.
  • modules and units involved in the described embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner. Among them, the name of the module or unit does not constitute a limitation on the module or unit itself under certain circumstances.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Des modes de réalisation de la présente invention concernent un procédé et un dispositif d'interfonctionnement de données, ainsi qu'un terminal et un support de stockage associés. Le procédé d'interfonctionnement de données consiste : à recevoir, par une deuxième base de données, des données de synchronisation transmises en temps réel par une première base de données au moyen d'un premier ensemble de tuyaux de données (TPIPE); et à commander la deuxième base de données pour transmettre, à une base de données esclave de la deuxième base de données, et au moyen d'un deuxième ensemble de tuyaux de données (TPIPE), des données complètes dans la deuxième base de données, le premier ensemble de tuyaux de données (TPIPE) servant à transmettre de façon ordonnée les données de synchronisation, et le deuxième ensemble de tuyaux de données (TPIPE) servant à transmettre de façon ordonnée les données complètes.
PCT/CN2020/076050 2019-02-27 2020-02-20 Procédé et dispositif d'interfonctionnement de données, terminal et support de stockage WO2020173381A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910146908.6A CN110083657A (zh) 2019-02-27 2019-02-27 数据互通方法、装置、终端及存储介质
CN201910146908.6 2019-02-27

Publications (1)

Publication Number Publication Date
WO2020173381A1 true WO2020173381A1 (fr) 2020-09-03

Family

ID=67413063

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/076050 WO2020173381A1 (fr) 2019-02-27 2020-02-20 Procédé et dispositif d'interfonctionnement de données, terminal et support de stockage

Country Status (2)

Country Link
CN (1) CN110083657A (fr)
WO (1) WO2020173381A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110083657A (zh) * 2019-02-27 2019-08-02 北京字节跳动网络技术有限公司 数据互通方法、装置、终端及存储介质
CN111858762A (zh) * 2020-07-21 2020-10-30 中冶南方工程技术有限公司 一种全流程质量管理系统的数据存储方法和系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105760552A (zh) * 2016-03-25 2016-07-13 北京奇虎科技有限公司 数据管理方法和装置
CN107734017A (zh) * 2017-09-30 2018-02-23 平安科技（深圳）有限公司 数据服务方法及系统
EP3312743A1 (fr) * 2016-10-24 2018-04-25 Samsung SDS Co., Ltd. Procédé et système de stockage de résultat de requête dans un serveur distribué
CN109325072A (zh) * 2018-07-12 2019-02-12 浪潮软件集团有限公司 一种基于CMSP实现跨网闸Oracle数据库同步的方法及装置
CN110083657A (zh) * 2019-02-27 2019-08-02 北京字节跳动网络技术有限公司 数据互通方法、装置、终端及存储介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10509801B2 (en) * 2013-01-28 2019-12-17 Arris Enterprises Llc Selective data synchronization and transfer to remote distributed database servers
CN105635074A (zh) * 2014-11-06 2016-06-01 中兴通讯股份有限公司 实现数据同步的方法、服务器、客户端和系统
CN105574109A (zh) * 2015-12-10 2016-05-11 青岛海信传媒网络技术有限公司 一种数据库同步方法、同步设备及系统
CN107391628B (zh) * 2017-06-30 2021-04-09 北京奇虎科技有限公司 数据同步方法及装置
CN109299088A (zh) * 2018-08-22 2019-02-01 中国平安人寿保险股份有限公司 海量数据存储方法、装置、存储介质及电子设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105760552A (zh) * 2016-03-25 2016-07-13 北京奇虎科技有限公司 数据管理方法和装置
EP3312743A1 (fr) * 2016-10-24 2018-04-25 Samsung SDS Co., Ltd. Procédé et système de stockage de résultat de requête dans un serveur distribué
CN107734017A (zh) * 2017-09-30 2018-02-23 平安科技（深圳）有限公司 数据服务方法及系统
CN109325072A (zh) * 2018-07-12 2019-02-12 浪潮软件集团有限公司 一种基于CMSP实现跨网闸Oracle数据库同步的方法及装置
CN110083657A (zh) * 2019-02-27 2019-08-02 北京字节跳动网络技术有限公司 数据互通方法、装置、终端及存储介质

Also Published As

Publication number Publication date
CN110083657A (zh) 2019-08-02

Similar Documents

Publication Publication Date Title
WO2020173248A1 (fr) Procédé et dispositif de synchronisation de données, terminal et support de stockage
WO2020221190A1 (fr) Procédé de synchronisation d'état d'applet, dispositif, et support d'enregistrement informatique
WO2021013242A1 (fr) Procédé de lancement d'un mini-programme, dispositif et support de stockage informatique
CN110083660A (zh) 一种同步数据的方法、装置、介质和电子设备
WO2023273725A1 (fr) Procédé et appareil de téléchargement de fichier, support de stockage et dispositif électronique
WO2020020202A1 (fr) Procédé et appareil de gestion de compte d'un programme d'application et dispositif et support
WO2021190557A1 (fr) Procédé et appareil d'interaction et dispositif électronique.
WO2021013247A1 (fr) Procédé et dispositif d'exécution d'appliquette, et support d'enregistrement informatique
WO2021082649A1 (fr) Procédé et appareil de mise à jour de liste, support lisible et dispositif électronique
WO2020168878A1 (fr) Procédé et dispositif de mise en cache de données, terminal et support d'enregistrement
WO2021190129A1 (fr) Procédé et dispositif de traitement de page, dispositif électronique et support de stockage lisible par ordinateur
WO2021073205A1 (fr) Procédé et appareil de traitement de vidéos, support de stockage et dispositif électronique
WO2023284387A1 (fr) Procédé, appareil et système de formation de modèle basés sur un apprentissage fédéré, dispositif et support
WO2021169863A1 (fr) Procédé et appareil de traitement de données de conférence multimédia, et dispositif électronique
WO2020173381A1 (fr) Procédé et dispositif d'interfonctionnement de données, terminal et support de stockage
CN109471647B (zh) 一种数据的更新方法、装置、电子设备及可读介质
WO2021088671A1 (fr) Procédé d'appel d'une fonction de terminal, dispositif et support de stockage informatique
CN110633046A (zh) 一种分布式系统的存储方法、装置、存储设备及存储介质
CN111163336B (zh) 视频资源推送方法、装置、电子设备及计算机可读介质
CN107817962B (zh) 一种远程控制方法、装置、控制服务器及存储介质
CN108470008B (zh) 串口数据读写方法、装置、计算机设备和存储介质
WO2020220777A1 (fr) Procédé de transfert d'informations, dispositif, équipement et support
US20240146978A1 (en) Functional component loading method and data processing method for video live-streaming, and device
US20230385080A1 (en) Method and apparatus for presenting information on lock screen interface, terminal, and storage medium
WO2021147466A1 (fr) Procédé et appareil de traitement de message et dispositif électronique

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20762724

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 21/01/2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20762724

Country of ref document: EP

Kind code of ref document: A1