CN115396453A - Data multistage synchronization method, device, equipment and medium - Google Patents

Data multistage synchronization method, device, equipment and medium Download PDF

Info

Publication number
CN115396453A
CN115396453A CN202211000024.8A CN202211000024A CN115396453A CN 115396453 A CN115396453 A CN 115396453A CN 202211000024 A CN202211000024 A CN 202211000024A CN 115396453 A CN115396453 A CN 115396453A
Authority
CN
China
Prior art keywords
synchronization
service data
synchronized
data
protocol
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211000024.8A
Other languages
Chinese (zh)
Inventor
董辉
张德林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
DBAPPSecurity Co Ltd
Original Assignee
DBAPPSecurity Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by DBAPPSecurity Co Ltd filed Critical DBAPPSecurity Co Ltd
Priority to CN202211000024.8A priority Critical patent/CN115396453A/en
Publication of CN115396453A publication Critical patent/CN115396453A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/18—Multiprotocol handlers, e.g. single devices capable of handling multiple protocols

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a data multilevel synchronization method, a device, equipment and a medium, comprising the following steps: determining a synchronization protocol based on the instantaneity requirement and the data volume of the service data to be synchronized, and determining the synchronization direction of the service data to be synchronized; wherein, the synchronization direction is up-level synchronization and/or down-level synchronization; and packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that the service data packet in the cache is synchronized to a second target platform when the first target platform synchronizes data to the second target platform. In this way, multi-level synchronization of data can be achieved, and synchronization efficiency is improved.

Description

Data multistage synchronization method, device, equipment and medium
Technical Field
The present application relates to the field of data synchronization technologies, and in particular, to a method, an apparatus, a device, and a medium for multilevel synchronization of data.
Background
In a service scenario, data synchronization is often required, and with continuous development of services, a requirement for performing multi-level synchronization on data gradually arises.
Disclosure of Invention
In view of this, an object of the present application is to provide a method, an apparatus, a device and a medium for multilevel data synchronization, which can achieve multilevel data synchronization and improve synchronization efficiency. The specific scheme is as follows:
in a first aspect, the present application discloses a data multilevel synchronization method, including:
determining a synchronization protocol based on the instantaneity requirement and the data volume of the service data to be synchronized, and determining the synchronization direction of the service data to be synchronized; wherein, the synchronization direction is up-level synchronization and/or down-level synchronization;
and packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that the service data packet in the cache is synchronized to a second target platform when the first target platform synchronizes data to the second target platform.
Optionally, the determining a synchronization protocol based on the instantaneity requirement of the service data to be synchronized and the data volume includes:
if the instantaneity requirement of the service data to be synchronized meets a preset high instantaneity condition and the data volume is smaller than a first preset data volume threshold value, determining that the synchronization protocol is an http protocol;
if the instantaneity requirement of the service data to be synchronized meets a preset instantaneity medium condition and the data volume is greater than a second preset data volume threshold value, determining that the synchronization protocol is a file transfer protocol;
and if the instantaneity requirement of the service data to be synchronized meets a preset low instantaneity condition and the data volume is greater than a third preset data volume threshold value, determining that the synchronization protocol is a queue protocol.
Optionally, the packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in the first target platform according to the synchronization direction includes:
if the synchronous protocol is determined to be an http protocol and the synchronous direction is towards the upper stage, reading the service data to be synchronized from the database of the platform of the current stage based on the write version number, packaging the service data to be synchronized into a service data packet, and transmitting the service data packet and the read version number to the first target platform, so that the first target platform stores the service data packet into the database of the first target platform;
acquiring data to be written in a subordinate and a data version of the data to be written in the subordinate, which are returned by the first target platform based on the read version number, updating the write version number, and updating the read version number into the data version; and the write version number represents the initial position of the service data to be synchronized.
Optionally, the packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction includes:
if the synchronization protocol is determined to be a queue protocol and the synchronization direction is the upper-level synchronization, the service data to be synchronized is packaged into a service data packet based on the queue protocol, and the service data packet is transmitted to a first target theme of the first target platform;
and if the synchronization protocol is determined to be a queue protocol and the synchronization direction is the lower-level synchronization, the service data to be synchronized is packaged into a service data packet based on the queue protocol, and the service data packet is stored in a second target theme of the current-level platform, so that the first target platform can pull the service data packet from the second target theme and store the service data packet in a first target theme of the first target platform.
Optionally, the packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction includes:
and if the synchronization protocol is determined to be a file transfer protocol, packaging the service data to be synchronized into a service data packet based on the file transfer protocol, and synchronizing the service data packet to a target file in a first target platform.
Optionally, the method further includes:
and monitoring a service event, and acquiring the data to be synchronized from the service event based on a preset rule.
Optionally, synchronizing the service data packet to the first target platform according to the synchronization direction includes:
synchronizing the service data packet and the platform unique code of the platform to a first target platform according to a synchronization direction;
correspondingly, when the first target platform synchronizes data to the second target platform, the service data packet in the cache and the platform unique code of the first target platform are synchronized to the second target platform.
In a second aspect, the present application discloses a data multi-level synchronization apparatus, including:
the synchronization protocol determining module is used for determining a synchronization protocol based on the instantaneity requirement and the data volume of the service data to be synchronized;
a synchronization direction determining module, configured to determine a synchronization direction of the service data to be synchronized; wherein, the synchronization direction is up-level synchronization and/or down-level synchronization;
and the service data synchronization module is used for packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that the service data packet in the cache is synchronized to a second target platform when the first target platform synchronizes data to the second target platform.
In a third aspect, the present application discloses an electronic device comprising a memory and a processor, wherein:
the memory is used for storing a computer program;
the processor is used for executing the computer program to realize the data multilevel synchronization method.
In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned data multi-level synchronization method.
Therefore, the method and the device for synchronizing the business data determine a synchronization protocol based on the instantaneity requirement and the data volume of the business data to be synchronized, and determine the synchronization direction of the business data to be synchronized; wherein, the synchronization direction is to the upper synchronization and/or the lower synchronization; and packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that the service data packet in the cache is synchronized to a second target platform when the first target platform synchronizes data to the second target platform. That is, when the synchronization protocol is determined, the instantaneity requirement and the data volume of the service data to be synchronized are considered, the data to be synchronized with different instantaneity requirements and data volumes are respectively processed, and the intermediate platform directly synchronizes the synchronized service data packet to the subsequent platform, so that the packaging process of the intermediate platform is reduced, thus, the multi-stage synchronization of the data is realized, and the synchronization efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of a method for multi-level synchronization of data disclosed herein;
FIG. 2 is a flow chart of a specific multi-level synchronization of data disclosed herein;
FIG. 3 is a diagram illustrating a specific multi-level synchronization scenario disclosed herein;
FIG. 4 is a schematic diagram illustrating a specific data multilevel synchronization disclosed herein;
FIG. 5 is a schematic diagram of a data multistage synchronization apparatus according to the present disclosure;
fig. 6 is a block diagram of an electronic device disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In a service scenario, data synchronization is often required, and with continuous development of services, a requirement for performing multi-level synchronization on data gradually arises. Therefore, the embodiment of the application provides a data multistage synchronization scheme, which can realize data multistage synchronization and improve synchronization efficiency.
Referring to fig. 1, the embodiment of the present application discloses a data multistage synchronization method, including:
step S11: determining a synchronization protocol based on the instantaneity requirement and the data volume of the service data to be synchronized, and determining the synchronization direction of the service data to be synchronized; wherein the synchronization direction is up-level synchronization and/or down-level synchronization.
In a specific implementation manner, if the instantaneity requirement of the service data to be synchronized meets a preset high instantaneity condition and the data volume is smaller than a first preset data volume threshold value, determining that a synchronization protocol is an http protocol; if the instantaneity requirement of the service data to be synchronized meets the medium condition of preset instantaneity and the data volume is greater than a second preset data volume threshold value, determining that a synchronization protocol is a file transfer protocol; and if the instantaneity requirement of the service data to be synchronized meets a preset low instantaneity condition and the data volume is greater than a third preset data volume threshold value, determining that the synchronization protocol is a queue protocol. The queue protocol may be kafka protocol or the like. The instantaneity requirement meets a preset high instantaneity condition within a first preset processing time, for example, within 1 minute, the instantaneity requirement is greater than the first preset processing time, is less than or equal to a third processing time, meets a preset instantaneity medium condition, for example, more than 1 minute to 10 minutes, and if the instantaneity requirement is greater than the third preset processing time, a timeliness low condition is met, for example, greater than 10 minutes.
It should be noted that, in actual service, data is generally divided into scenes with high instantaneity, small data volume, high instantaneity, large data volume, and low instantaneity, and different synchronization protocols, that is, synchronization modes, are provided in the present application for the three scenes. For the high instantaneity and the large data volume, a file transmission protocol or a queue protocol can be selected according to the requirement, and the simultaneous storage of the large data volume does not accord with the logic and needs to be chosen in the instantaneity and the data volume. Besides, data are shunted through different synchronization protocols, the embodiment of the application can add read-write efficiency limitation to data with high real-time performance, data which do not meet read-write efficiency can be rejected, and the problem of other data real-time performance caused by certain data problems is prevented. Specifically, in the process of synchronization through the http protocol, the data processing time of the first target platform may be monitored, and if the preset time is exceeded, the remaining data to be synchronized is discarded. That is, the embodiment of the application improves the real-time performance of data through data distribution and read-write efficiency limitation.
For example, according to the scheme provided by the embodiment of the application, the real-time performance is high, the data volume is small, the data can be synchronously completed within 1 minute, the timeliness is high, the data volume is large, the data can be synchronously completed within about 10 minutes, the timeliness is not high, and when the data volume is extremely large, delay can be generated according to different data synchronization time.
In addition, in an embodiment, a service event may be monitored, and the data to be synchronized may be obtained from the service event based on a preset rule.
Step S12: and packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that the service data packet in the cache is synchronized to a second target platform when the first target platform synchronizes data to the second target platform.
In a specific implementation manner, in the embodiment of the present application, two platforms can perform unidirectional network communication, and in the embodiment of the present application, multi-level synchronization of data can be realized in a case where a synchronous network is only unidirectional, and the implementation idea is that one side of the network communication pushes and pulls data, the other side issues data that should be pulled, and meanwhile, data that needs to be synchronized is provided with an ordered version number.
In an embodiment, if it is determined that the synchronization protocol is an http protocol and the synchronization direction is an upper-level synchronization, reading the service data to be synchronized from the database of the local-level platform based on a write version number, packaging the service data to be synchronized into a service data packet, and transmitting the service data packet and the read version number to the first target platform, so that the first target platform stores the service data packet in the database of the first target platform; acquiring data to be written in a subordinate and a data version of the data to be written in the subordinate, which are returned by the first target platform based on the read version number, updating the write version number, and updating the read version number into the data version; and the write version number represents the initial position of the service data to be synchronized.
For example, the initial write version number corresponds to a first piece of data, the first piece of data, a second piece of data, and a third piece of data are synchronized at a time, and after synchronization, the write version number is updated to correspond to a fourth piece of data.
Therefore, the transmission of data between two intercommunicating platforms is realized by the change of reading version number and writing version number, and the synchronization can be carried out in a network of one-way communication.
In one embodiment, if it is determined that a synchronization protocol is a queuing protocol and the synchronization direction is an upper-level synchronization, the service data to be synchronized is packaged into a service data packet based on the queuing protocol, and the service data packet is transmitted to a first target topic of the first target platform; and if the synchronization protocol is determined to be a queue protocol and the synchronization direction is the lower-level synchronization, the service data to be synchronized is packaged into a service data packet based on the queue protocol, and the service data packet is stored in a second target theme of the current-level platform, so that the first target platform can pull the service data packet from the second target theme and store the service data packet in a first target theme of the first target platform. With the offset of kafka as the read version number and the write version number.
In an embodiment, if it is determined that the synchronization protocol is a file transfer protocol, the service data to be synchronized is packed into a service data packet based on the file transfer protocol, and the service data packet is synchronized to a target file in a first target platform. The file name is used as the version number, the date is used as the file name, the embodiment of the application stores the data to be synchronized in the file named by the date, and the service data packet is obtained based on the write version number, namely the current write version number is used as the initial file, all the files which are not synchronized and the read version number are transmitted to the first target platform, so that the first target platform stores the service data packet in the target file; and obtaining the file to be written in the subordinate and the new read version number returned by the first target platform based on the read version number, and updating the write version number.
Further, in one embodiment, the first target platform may modify the service data packet based on a preset hook function, and then synchronize to the second target platform.
It should be noted that, in a data synchronization scenario, a synchronization sequence often occurs or a synchronization efficiency is problematic, and a problem in sequence directly affects the service itself, even if a short-term sequence is problematic, the understanding of the synchronized party on the data is also affected. The synchronization efficiency is a core problem, and the design of subsequent requirements is influenced. In the prior art, most of two devices are synchronized, namely, A- > B (namely, A is synchronized to B), and a lot of people understand that A- > B can be used for B- > C, but the significance of multi-level synchronization is that B transmits the data of A to C instead of packing the data of A once again, so that the problems of complex synchronization relationship and disordered data sequence can be avoided. By adopting the scheme, through transparent transmission, the packaging process of the intermediate platform is reduced, the time is saved, the efficiency is higher, and more time is saved between the multi-stage platforms. The embodiment of the application can perform some micro-processing in transparent transmission. For example, A- > B- > C, B to C is transparent transmission, and a scalable hook is added between B and C, and data from B to C can be modified through the hook. That is to say, the embodiment of the application can meet the synchronization requirement with high real-time performance and avoid the problem of secondary data packaging during data multi-stage synchronization.
In a specific embodiment, the service data packet and the platform unique code of the current-stage platform are synchronized to the first target platform according to a synchronization direction; correspondingly, when the first target platform synchronizes data to the second target platform, the service data packet in the cache and the platform unique code of the first target platform are synchronized to the second target platform. Therefore, the unique code of the platform is added, the propagation link can be recorded, and the synchronous cycle is prevented from occurring.
Further, in the embodiment of the present application, the service data packet may be synchronized to the cache corresponding to the synchronization protocol in the first target platform according to the synchronization direction through a synchronization dedicated thread, so that when the first target platform synchronizes data to the second target platform, the service data packet in the cache is synchronized to the second target platform. Therefore, when the real-time synchronization is realized, the other thread which is synchronously run is synchronized, the thread can normally carry out service operation when the synchronized service system is abnormal, and when the synchronized service system recovers to be normal, the compensation type synchronization of the part of service realizes fusing.
And when the platform health monitoring thread monitors that the first target platform is healthy, the service data packet can be synchronized to the cache corresponding to the synchronization protocol in the first target platform through the synchronization special thread according to the synchronization direction, so that the influence on the system performance caused by continuous synchronization attempts when the platform fails is avoided.
For example, referring to fig. 2, fig. 2 is a specific data multilevel synchronization flowchart disclosed in the embodiment of the present application. Monitoring service events, judging whether data needs to be synchronized, if so, packaging the service data, selecting a corresponding synchronization protocol, and synchronizing through a synchronization system. The synchronization protocol, namely the mode used during synchronization, comprises http, files and queues, and the synchronization direction comprises upward, downward and bidirectional synchronization; and, can realize the packer, while the business takes place, judge whether the business needs to pack into the data, confirm the direction of synchronization and store predefined synchronous way. And the service logic for realizing the unpacking device and unpacking the data transmission is opposite to that of the packing device. The implementation filter performs some operations on the transparent transmission data; and the monitor is realized, data is preprocessed, and data is actually put in a warehouse and unpacked.
First two platforms can carry out one-way network UNICOM two by two, this application embodiment can only realize the multistage synchronization of data in the one-way situation that can of synchronous network, realizes that the thinking is that one side that the network leads to carries out the propelling movement and pulls off the data, and another side issues the data that should be pulled, and the data that need the synchronization will be printed an ordered version number simultaneously.
Http transport is introduced first below: when the lower platform generates a write data version number, i.e. a write version number, for example, an initial write version number [ 0 ], and is communicated with the upper platform, the upper platform returns the latest data version of the data to be synchronized, and stores the latest data version in the local, and the data version is recorded as a read version number [ 0 ]. Respectively, that the next time data of more than 0 version should be written, the data of more than 0 version should be read from the upper stage platform. And then, data is produced and stored, the service implementation has event sending, wherein the event sending comprises service data, the event can be monitored, the concerned data is stored in a database through a corresponding rule according to a pre-declared rule, the main key of the database is self-increased, other protocols are stored in storage positions corresponding to other protocols, and the unique code of the platform at the current level is recorded. Therefore, the upper and lower platforms can accumulate a part of data to be synchronized, the embodiment of the application monitors that the data is not directly transmitted, and the data synchronization is stripped from the service, so that the performance of the system is prevented from being influenced. And then, synchronizing the data, acquiring a certain amount of data by a lower level through a writing version number (0), packaging the data if the acquired data is not empty, transmitting the data to the upper level when communicating with the upper level, simultaneously transmitting the read version number (0) to the upper level, warehousing the data after the upper level receives the data, executing asynchronous transparent transmission logic, returning the data to be written into the lower level and a corresponding data version according to the reading version number (0), processing the warehoused data firstly after the lower level receives the data normally, updating the writing version to be the last version written last time, and updating the reading version to be the data version returned by the upper level. The asynchronous transparent transmission logic is as follows: if the data is sent from the lower level, the data is sent upwards, and if the data does not have the upper level, the data is ended. If the superior exists, the data is stored in a database, and the unique code of the platform at the current level is added on a propagation link. The next current stage platform as the lower stage can be synchronized to the upper stage according to the logic.
Furthermore, the kafka synchronization has no concept of real version number under the introduction of analogy, the offset of kafka is used as the version number, each stage platform creates the subject of own received data, the subject which needs to be synchronized to the stage platform is sent to the stage platform, the subject which needs to be synchronized is also sent to the stage platform, and the monitoring of the subject is the subject which is self and the lower stage of the subject. For example, data a needs to be sent up, and then is sent to a higher-level theme, and the higher level receives and puts in storage; the upper level sends the subject to be sent to the lower level for fetching. Meanwhile, the concept of transparent transmission is also provided, and the transparent transmission is stored in kafka.
Further, the introduction of the analogy is to download file, the date is used as the file name, the file name is used as the version number, the generated data is stored in the file with the date as the file name, when the file is synchronized upwards, the file which is not synchronized is transferred upwards by one copy, and when the file is synchronized downwards, the file which is not synchronized is taken out by the lower level. The data can be stored in a warehouse and then stored in a file, and the transparent transmission logic is also arranged in the file.
Therefore, through the change of the read version and the write version, the data transmission between the two intercommunicating platforms is realized, and the synchronization can be carried out in a unidirectional communication network; the synchronization capability is enhanced through the transparent transmission capability. The method supports one-time collection and multi-level synchronization, and reduces the packaging process of the intermediate platform during the multi-level synchronization of the data, thereby improving the efficiency; moreover, the transparent transmission opens the transparent transmission modification function, strengthens the transparent transmission enabling, and the platform receiving the data can selectively modify and reprocess the data when spreading synchronization, thereby facilitating the relationship conversion among the requirements; through the diversity of the synchronization mode, the method realizes the stable performance, the strong expandability and the multiple synchronization schemes under various scenes, and the scenes which can be used can be selected according to the requirements and can be switched at will.
For example, referring to fig. 3, fig. 3 is a schematic diagram of a specific multi-level synchronization scenario, where a, B, C, and D denote multiple platforms, and arrows denote data synchronization directions, and further, referring to fig. 4, fig. 4 is a schematic diagram of a specific multi-level synchronization of data disclosed in an embodiment of the present application. D is used as a lower platform of B, 1, data is written into B, a reading id (namely a reading version number) is transmitted, 3 and D return data according to the reading id, 4, the latest id of an upper stage is returned at the same time, 5, B modifies the reading id and the writing id (namely the writing version number), B transmits the data to A thoroughly, and D-B represents the data synchronization process from D to B. Wherein, C can read the data of D through A, and can also synchronize the data to A, from A to D.
Therefore, the embodiment of the application determines a synchronization protocol based on the instantaneity requirement and the data volume of the service data to be synchronized, and determines the synchronization direction of the service data to be synchronized; wherein, the synchronization direction is up-level synchronization and/or down-level synchronization; and packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that the service data packet in the cache is synchronized to a second target platform when the first target platform synchronizes data to the second target platform. That is, when the synchronization protocol is determined, the instantaneity requirement and the data volume of the service data to be synchronized are considered, the data to be synchronized with different instantaneity requirements and data volumes are respectively processed, and the intermediate platform directly synchronizes the synchronized service data packet to the subsequent platform, so that the packaging process of the intermediate platform is reduced, thus, the multi-stage synchronization of the data is realized, and the synchronization efficiency is improved.
Referring to fig. 5, an embodiment of the present application discloses a data multilevel synchronization apparatus, including:
a synchronization protocol determining module 11, configured to determine a synchronization protocol based on an instantaneity requirement of service data to be synchronized and a data volume;
a synchronization direction determining module 12, configured to determine a synchronization direction of the service data to be synchronized; wherein, the synchronization direction is to the upper synchronization and/or the lower synchronization;
a service data synchronization module 13, configured to package the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronize the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that when the first target platform synchronizes data to a second target platform, the service data packet in the cache is synchronized to the second target platform.
Therefore, the embodiment of the application determines a synchronization protocol based on the instantaneity requirement and the data volume of the service data to be synchronized, and determines the synchronization direction of the service data to be synchronized; wherein, the synchronization direction is to the upper synchronization and/or the lower synchronization; and packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that the service data packet in the cache is synchronized to a second target platform when the first target platform synchronizes data to the second target platform. That is, when the synchronization protocol is determined, the instantaneity requirement and the data volume of the service data to be synchronized are considered, the data to be synchronized with different instantaneity requirements and data volumes are respectively processed, and the intermediate platform directly synchronizes the synchronized service data packet to the subsequent platform, so that the packaging process of the intermediate platform is reduced, thus, the multi-stage synchronization of the data is realized, and the synchronization efficiency is improved.
In an embodiment, if the instantaneity requirement of the service data to be synchronized meets a preset high instantaneity condition and the data volume is smaller than a first preset data volume threshold, the synchronization protocol determining module determines that the synchronization protocol is an http protocol; if the instantaneity requirement of the service data to be synchronized meets a preset instantaneity medium condition and the data volume is greater than a second preset data volume threshold value, determining that the synchronization protocol is a file transfer protocol; and if the instantaneity requirement of the service data to be synchronized meets a preset low instantaneity condition and the data volume is greater than a third preset data volume threshold value, determining that the synchronization protocol is a queue protocol.
Further, the service data synchronization module 13 is specifically configured to:
if the synchronous protocol is determined to be an http protocol and the synchronous direction is towards the upper stage, reading the service data to be synchronized from the database of the platform of the current stage based on the write version number, packaging the service data to be synchronized into a service data packet, and transmitting the service data packet and the read version number to the first target platform, so that the first target platform stores the service data packet into the database of the first target platform; acquiring data to be written in a subordinate and a data version of the data to be written in the subordinate, which are returned by the first target platform based on the read version number, updating the write version number, and updating the read version number into the data version; and the write version number represents the initial position of the service data to be synchronized.
If the synchronization protocol is determined to be a queue protocol and the synchronization direction is the upper-level synchronization, the service data to be synchronized is packaged into a service data packet based on the queue protocol, and the service data packet is transmitted to a first target theme of the first target platform; and if the synchronization protocol is determined to be a queue protocol and the synchronization direction is the lower-level synchronization, the service data to be synchronized is packaged into a service data packet based on the queue protocol, and the service data packet is stored in a second target theme of the current-level platform, so that the first target platform can pull the service data packet from the second target theme and store the service data packet in a first target theme of the first target platform.
And if the synchronization protocol is determined to be a file transfer protocol, packaging the service data to be synchronized into a service data packet based on the file transfer protocol, and synchronizing the service data packet to a target file in a first target platform.
Further, the device comprises a module for acquiring the data to be synchronized, which is used for monitoring the business events and acquiring the data to be synchronized from the business events based on the preset rules.
Further, the service data synchronization module 13 is specifically configured to synchronize the service data packet and the platform unique code of the current-stage platform to the first target platform according to a synchronization direction; correspondingly, when the first target platform synchronizes data to the second target platform, the service data packet in the cache and the platform unique code of the first target platform are synchronized to the second target platform.
Referring to fig. 6, an embodiment of the present application discloses an electronic device 20, which includes a processor 21 and a memory 22; wherein, the memory 22 is used for storing computer programs; the processor 21 is configured to execute the computer program, and the data multi-level synchronization method disclosed in the foregoing embodiments.
For the specific process of the data multistage synchronization method, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
The memory 22 is used as a carrier for resource storage, and may be a read-only memory, a random access memory, a magnetic disk or an optical disk, and the storage mode may be a transient storage mode or a permanent storage mode.
In addition, the electronic device 20 further includes a power supply 23, a communication interface 24, an input-output interface 25, and a communication bus 26; the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and a communication protocol followed by the communication interface is any communication protocol applicable to the technical solution of the present application, and is not specifically limited herein; the input/output interface 25 is configured to obtain external input data or output data to the outside, and a specific interface type thereof may be selected according to a specific application requirement, which is not specifically limited herein.
Further, the present application also discloses a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the data multi-level synchronization method disclosed in the foregoing embodiments.
For the specific process of the data multistage synchronization method, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
In the present specification, the embodiments are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same or similar parts between the embodiments are referred to each other. The device disclosed in the embodiment corresponds to the method disclosed in the embodiment, so that the description is simple, and the relevant points can be referred to the description of the method part.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The foregoing describes in detail a method, an apparatus, a device, and a medium for data multilevel synchronization provided by the present application, and specific examples are applied herein to explain the principles and implementations of the present application, and the descriptions of the foregoing examples are only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims (10)

1. A method for multi-level synchronization of data, comprising:
determining a synchronization protocol based on the instantaneity requirement and the data volume of the service data to be synchronized, and determining the synchronization direction of the service data to be synchronized; wherein, the synchronization direction is up-level synchronization and/or down-level synchronization;
and packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that the service data packet in the cache is synchronized to a second target platform when the first target platform synchronizes data to the second target platform.
2. The data multistage synchronization method according to claim 1, wherein the determining a synchronization protocol based on the instantaneity requirement and the data volume of the service data to be synchronized comprises:
if the instantaneity requirement of the service data to be synchronized meets a preset high instantaneity condition and the data volume is smaller than a first preset data volume threshold value, determining that the synchronization protocol is an http protocol;
if the instantaneity requirement of the service data to be synchronized meets the medium condition of preset instantaneity and the data volume is greater than a second preset data volume threshold value, determining that a synchronization protocol is a file transfer protocol;
and if the instantaneity requirement of the service data to be synchronized meets a preset low instantaneity condition and the data volume is greater than a third preset data volume threshold value, determining that the synchronization protocol is a queue protocol.
3. The multi-level data synchronization method according to claim 2, wherein the step of packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction comprises:
if the synchronous protocol is determined to be an http protocol and the synchronous direction is towards the upper stage, reading the service data to be synchronized from the database of the platform of the current stage based on the write version number, packaging the service data to be synchronized into a service data packet, and transmitting the service data packet and the read version number to the first target platform, so that the first target platform stores the service data packet into the database of the first target platform;
acquiring data to be written in a subordinate and a data version of the data to be written in the subordinate, which are returned by the first target platform based on the read version number, updating the write version number, and updating the read version number into the data version; and the write version number represents the initial position of the service data to be synchronized.
4. The multi-level data synchronization method according to claim 2, wherein the step of packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction comprises:
if the synchronization protocol is determined to be a queuing protocol and the synchronization direction is the upper synchronization, packing the service data to be synchronized into a service data packet based on the queuing protocol, and transmitting the service data packet to a first target subject of the first target platform;
and if the synchronization protocol is determined to be a queue protocol and the synchronization direction is the lower-level synchronization, the service data to be synchronized is packaged into a service data packet based on the queue protocol, and the service data packet is stored in a second target theme of the current-level platform, so that the first target platform can pull the service data packet from the second target theme and store the service data packet in a first target theme of the first target platform.
5. The data multilevel synchronization method according to claim 2, wherein the step of packing the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction comprises:
and if the synchronization protocol is determined to be a file transfer protocol, packaging the service data to be synchronized into a service data packet based on the file transfer protocol, and synchronizing the service data packet to a target file in a first target platform.
6. The method for multi-level synchronization of data according to claim 1, further comprising:
and monitoring a business event, and acquiring the data to be synchronized from the business event based on a preset rule.
7. The multi-level data synchronization method according to any one of claims 1 to 6, wherein synchronizing the service data packet to the first target platform according to a synchronization direction comprises:
synchronizing the service data packet and the platform unique code of the platform to a first target platform according to a synchronization direction;
correspondingly, when the first target platform synchronizes data to a second target platform, the service data packet in the cache and the platform unique code of the first target platform are synchronized to the second target platform.
8. A data multi-level synchronization apparatus, comprising:
the synchronization protocol determining module is used for determining a synchronization protocol based on the instantaneity requirement and the data volume of the service data to be synchronized;
a synchronization direction determining module, configured to determine a synchronization direction of the service data to be synchronized; wherein, the synchronization direction is to the upper synchronization and/or the lower synchronization;
and the service data synchronization module is used for packaging the service data to be synchronized into a service data packet based on the synchronization protocol, and synchronizing the service data packet to a cache corresponding to the synchronization protocol in a first target platform according to a synchronization direction, so that the service data packet in the cache is synchronized to a second target platform when the first target platform synchronizes data to the second target platform.
9. An electronic device comprising a memory and a processor, wherein:
the memory is used for storing a computer program;
the processor for executing the computer program to implement the data multi-level synchronization method of any one of claims 1 to 7.
10. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the data multi-level synchronization method of any one of claims 1 to 7.
CN202211000024.8A 2022-08-19 2022-08-19 Data multistage synchronization method, device, equipment and medium Pending CN115396453A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211000024.8A CN115396453A (en) 2022-08-19 2022-08-19 Data multistage synchronization method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211000024.8A CN115396453A (en) 2022-08-19 2022-08-19 Data multistage synchronization method, device, equipment and medium

Publications (1)

Publication Number Publication Date
CN115396453A true CN115396453A (en) 2022-11-25

Family

ID=84121485

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211000024.8A Pending CN115396453A (en) 2022-08-19 2022-08-19 Data multistage synchronization method, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN115396453A (en)

Similar Documents

Publication Publication Date Title
CN111818136B (en) Data processing method, device, electronic equipment and computer readable medium
CN107222530A (en) Service asynchronous exchange method, equipment, system and computer-readable recording medium
US10708378B2 (en) Data processing method and apparatus, server, and controller
CN110502572A (en) A kind of method and server that data are synchronous
CN112800017A (en) Distributed log collection method, device, medium and electronic equipment
CN109783573A (en) The method of data synchronization and terminal of multichannel push
CN111294235A (en) Data processing method, device, gateway and readable storage medium
JP2012257122A (en) Vehicle controller and vehicle control system
CN103646015B (en) Transmission, the method and system for receiving and transmitting XML message
CN115396453A (en) Data multistage synchronization method, device, equipment and medium
CN111786859A (en) Packet loss detection method, device and storage medium
US20060064442A1 (en) System and method for adaptively changing application server processing power based on data volume
CN108289165A (en) A kind of implementation method, device and terminal device controlling camera based on mobile phone
JPH08292922A (en) Network management device
CN112217669B (en) Service data transmission method, device, terminal equipment and storage medium
US20200162284A1 (en) Management device, communication system, management method, and computer readable medium
CN112002115B (en) Data acquisition method and data processor
CN117997802B (en) Network jitter detection method, device, equipment and medium
CN114158067B (en) Method and device for data transmission in private network
EP4064589A1 (en) Ethernet time synchronization method and apparatus
CN115269261A (en) System switching method and device, electronic equipment and computer readable medium
CN101651571B (en) Analytic method and equipment for ports
CN108255770B (en) Processing method based on 1394 bus event message response mechanism
EP1220511A2 (en) Memory management for packet storage
CN116049174A (en) Electric power internet of things node information forwarding method and device and wireless node

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination