CN116894071A - Real-time parallel synchronization method and system for cross database - Google Patents
Real-time parallel synchronization method and system for cross database Download PDFInfo
- Publication number
- CN116894071A CN116894071A CN202310818394.0A CN202310818394A CN116894071A CN 116894071 A CN116894071 A CN 116894071A CN 202310818394 A CN202310818394 A CN 202310818394A CN 116894071 A CN116894071 A CN 116894071A
- Authority
- CN
- China
- Prior art keywords
- data
- data source
- synchronization
- static
- dynamic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 230000003068 static effect Effects 0.000 claims abstract description 61
- 230000001360 synchronised effect Effects 0.000 claims abstract description 41
- 230000003203 everyday effect Effects 0.000 claims 1
- 238000012360 testing method Methods 0.000 abstract description 15
- 238000003860 storage Methods 0.000 description 7
- 238000004590 computer program Methods 0.000 description 6
- 238000013500 data storage Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 238000010923 batch production Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
- G06F11/2023—Failover techniques
- G06F11/2028—Failover techniques eliminating a faulty processor or activating a spare
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/221—Column-oriented storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a real-time parallel synchronization method crossing databases, belonging to the field of satellite testing; the method comprises the following steps: registering a plurality of dynamic data sources; synchronizing data from a dynamic data source to a static data source, and recording a synchronization position; acquiring a query instruction; and according to the query instruction, querying the data from the corresponding synchronous position of the static data source. The invention also discloses a real-time parallel synchronization system crossing the database. The invention combines static data sources with a plurality of dynamic data sources for mixed deployment and supports dynamic and expandable updating of the dynamic data sources.
Description
Technical Field
The invention relates to the field of satellite testing, in particular to a real-time parallel synchronization method and system for a cross database.
Background
In the satellite research and development process, the integrity and the correctness of data storage are required to be ensured before the satellite is transmitted into orbit, so that the current test data are timely and effectively analyzed before the satellite is transmitted. Because a large amount of historical telemetry and remote control data and historical alarm data are required to be stored in the satellite test process, and meanwhile, the received data are large in difference and uneven in distribution, the requirement of query performance cannot be met by the simple row-type relational data storage, and therefore the query performance requirement of the satellite ground test is met by adopting a database deployment mode combining row-type storage and column-type query. How to synchronize data in real time and efficiently to a column database and meet the data query requirement of satellite testing in time is particularly critical.
Disclosure of Invention
The invention aims to provide a real-time parallel synchronization method and a system for crossing databases, which combine static data sources with a plurality of dynamic data sources for mixed deployment and support dynamic and expandable updating of the dynamic data sources.
In order to solve the technical problems, the invention provides a real-time parallel synchronization method across databases, which comprises the following steps:
registering a plurality of dynamic data sources;
synchronizing data from a dynamic data source to a static data source, and recording a synchronization position;
acquiring a query instruction;
and according to the query instruction, querying the data from the corresponding synchronous position of the static data source.
Preferably, the method for synchronizing data from a dynamic data source to a static data source specifically comprises the following steps:
acquiring a current dynamic data source interface according to the satellite unique mark of the dynamic data source;
the dynamic data source is synchronized to the static data source through the corresponding current dynamic data source interface.
Preferably, the data of the dynamic data source includes telemetry data, remote control instructions and telemetry alert data.
Preferably, the dynamic data source is synchronized to the static data source through the corresponding current dynamic data source interface, and specifically comprises the following steps:
obtaining a data table according to the data of the dynamic data source and the static data source;
according to the data table, the dynamic data source sequentially synchronizes telemetry data to the static data source through the corresponding current dynamic data source interface;
the dynamic data source synchronizes the remote control instruction and the telemetry alarm data to the static data source through the corresponding current dynamic data source interface.
Preferably, the method further comprises the following steps:
in the synchronization process, recording the synchronization state of the data table;
taking the maximum ID position in the data table as a synchronous ending position;
in the data synchronization process, whether the synchronization is completed or not is judged according to the synchronization ending position.
Preferably, the method further comprises the following steps:
after the first synchronization of the data is completed, judging whether the synchronization is completed or not according to the synchronization position, and detecting whether the data is missed.
Preferably, the dynamic data source synchronizes data to the static data source daily and updates the synchronization location.
Preferably, the data is queried from the corresponding synchronous position of the static data source, and the method specifically comprises the following steps:
inquiring whether corresponding data exists in the cache according to the satellite unique mark of the inquiring instruction;
if the corresponding data exists in the cache, downloading the data;
if the corresponding data does not exist in the cache, inquiring whether the corresponding data exists in the static data source;
if the corresponding data exists in the static data source, downloading the data;
if the static data source does not have corresponding data, synchronizing the data from the dynamic data source to the static data source every 5S until the corresponding data in the static data source is inquired.
Preferably, the data queried in the static data source is sent to the cache.
The invention also provides a real-time parallel synchronization system crossing databases, which comprises:
the registration module is used for registering a plurality of dynamic data sources;
the synchronization module is used for synchronizing data from the dynamic data source to the static data source and recording the synchronization position;
the acquisition module is used for acquiring the query instruction;
and the query module is used for querying data from the corresponding synchronous positions of the static data source according to the query instruction.
Compared with the prior art, the invention has the beneficial effects that:
a) Multiple data source combination:
the invention combines static data sources (column type) with a plurality of dynamic data sources (row type) for mixed deployment, and supports dynamic and expandable updating of the dynamic data sources, including the number of the dynamic data sources and the configuration of a database thereof. The data source configuration is dynamically added or deleted or modified by modifying the system configuration file, and then the restart service takes effect.
b) High reliability:
the invention adopts a cluster deployment scheme of haproxy+keep-alive-master-slave, and under the condition that the main server of the system is down, the standby server automatically preempts the drift IP, so that the service is continuously ensured to be uninterrupted, and the system stability can be greatly improved.
c) Real-time synchronization:
the invention introduces a database deployment mode combining line storage and column query to meet the requirement, can effectively realize read-write separation of data, improves the overall usability of the system, and becomes a necessary requirement for real-time synchronization of the line database and the column database. According to the invention, under the requirement of ensuring the query performance, data is actively synchronized once every 1 minute, so that the real-time query requirement of satellite test is basically met.
d) Introducing a caching mechanism:
a caching mechanism is introduced, considering that a user may perform the same or similar query needs multiple times. When the data is queried, if the cache is not hit, the data is recorded in the cache for storage, and when the next query is performed, the data can be hit from the cache and the query result can be directly returned from the cache to the user. Under the condition that the user inquires the miss cache, the data synchronization can wait for 5 seconds, then whether the inquiry is completed is detected, and after repeated detection, if the inquiry is completed, the data synchronization is continued, and the scheme improves the reliability of the data inquiry and the performance requirement of the data synchronization.
e) And (3) parallel execution:
considering the case where a plurality of satellites are simultaneously tested, it is required that data synchronization can be performed in parallel in order to improve the performance of synchronizing data. Under the condition that the physical resources (mainly CPU cores) of the server meet the requirements, an asynchronous multithreading synchronous mechanism is adopted, the parallel synchronous requirement of satellite testing is met, and the data synchronous performance requirement is greatly improved. The synchronization mechanism can wait for the synchronization position of each satellite to be recorded after all data are synchronized, so that new data can be conveniently synchronized from the recorded position continuously.
f) Full and delta synchronization:
in order to improve the stability of data synchronization, a data synchronization mechanism combining a full quantity mode and an increment mode is adopted. The incremental data synchronization aims at the data on the same day, and can meet the continuous updating synchronization of the data on the same day; for full data synchronization, two situations can be distinguished: the first is that the batch data are synchronized once when the system is started for the first time, and the second is that the data with the left synchronization are checked again for updating again when the system is started for one day. In general, the system meets the synchronous requirement of daily real-time update, also meets the stability requirement of data synchronization, and fully ensures the real-time and reliability requirements of data synchronization.
Drawings
The following describes the embodiments of the present invention in further detail with reference to the accompanying drawings.
FIG. 1 is a data synchronization with a multi-data source hybrid deployment;
FIG. 2 is a flow chart of data synchronization across databases;
fig. 3 is a flow chart of data synchronization for a single satellite data source.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. The present invention may be embodied in many other forms than those herein described, and those skilled in the art will readily appreciate that the present invention may be similarly embodied without departing from the spirit or essential characteristics thereof, and therefore the present invention is not limited to the specific embodiments disclosed below.
The terminology used in the one or more embodiments of the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the specification. As used in this specification, one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that, although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first may also be referred to as a second, and similarly, a second may also be referred to as a first, without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
The invention is described in further detail below with reference to the attached drawing figures:
the invention provides a real-time parallel synchronization method across databases, which comprises the following steps:
registering a plurality of dynamic data sources;
synchronizing data from a dynamic data source to a static data source, and recording a synchronization position;
acquiring a query instruction;
and according to the query instruction, querying the data from the corresponding synchronous position of the static data source.
Preferably, the method for synchronizing data from a dynamic data source to a static data source specifically comprises the following steps:
acquiring a current dynamic data source interface according to the satellite unique mark of the dynamic data source;
the dynamic data source is synchronized to the static data source through the corresponding current dynamic data source interface.
Preferably, the data of the dynamic data source includes telemetry data, remote control instructions and telemetry alert data.
Preferably, the dynamic data source is synchronized to the static data source through the corresponding current dynamic data source interface, and specifically comprises the following steps:
obtaining a data table according to the data of the dynamic data source and the static data source;
according to the data table, the dynamic data source sequentially synchronizes telemetry data to the static data source through the corresponding current dynamic data source interface;
the dynamic data source synchronizes the remote control instruction and the telemetry alarm data to the static data source through the corresponding current dynamic data source interface.
Preferably, the method further comprises the following steps:
in the synchronization process, recording the synchronization state of the data table;
taking the maximum ID position in the data table as a synchronous ending position;
in the data synchronization process, whether the synchronization is completed or not is judged according to the synchronization ending position.
Preferably, the method further comprises the following steps:
after the first synchronization of the data is completed, judging whether the synchronization is completed or not according to the synchronization position, and detecting whether the data is missed.
Preferably, the dynamic data source synchronizes data to the static data source daily and updates the synchronization location.
Preferably, the data is queried from the corresponding synchronous position of the static data source, and the method specifically comprises the following steps:
inquiring whether corresponding data exists in the cache according to the satellite unique mark of the inquiring instruction;
if the corresponding data exists in the cache, downloading the data;
if the corresponding data does not exist in the cache, inquiring whether the corresponding data exists in the static data source;
if the corresponding data exists in the static data source, downloading the data;
if the static data source does not have corresponding data, synchronizing the data from the dynamic data source to the static data source every 5S until the corresponding data in the static data source is inquired.
Preferably, the data queried in the static data source is sent to the cache.
The invention also provides a real-time parallel synchronization system crossing databases, which comprises:
the registration module is used for registering a plurality of dynamic data sources;
the synchronization module is used for synchronizing data from the dynamic data source to the static data source and recording the synchronization position;
the acquisition module is used for acquiring the query instruction;
and the query module is used for querying data from the corresponding synchronous positions of the static data source according to the query instruction.
In order to better illustrate the technical effects of the present invention, the present invention provides the following specific embodiments to illustrate the above technical flow:
embodiment 1, a real-time parallel synchronization method across databases:
aiming at satellite development in the batch production stage, the invention provides a method for cross-database combined deployment and real-time parallel synchronization to solve the problem of low batch data query performance. The cross-database hybrid deployment and data synchronization scheme is shown in fig. 1, and comprises a static data source and a plurality of expandable dynamic data sources, wherein each dynamic data source corresponds to test data of one satellite, and in order to meet the real-time data storage and query requirements of parallel testing of a plurality of satellites, real-time parallel data synchronization of the plurality of satellites becomes a key point.
The data synchronization process of the single satellite is shown in fig. 3, and the specific steps are as follows:
step 1: the table synchronization status is recorded. Can be used for judging whether the current table is finished synchronously;
step 2: synchronizing telemetry data. Acquiring a current dynamic data source interface according to a current satellite unique mark, circularly traversing a current data table to be synchronized, acquiring the maximum ID position of the current table as a synchronization ending position, judging whether synchronization is completed or not, and if so, continuing traversing the next table; otherwise, the segmentation is started to synchronize, 10 ten thousand pieces of data are synchronized once, and the current table synchronization state is updated after the synchronization is finished.
Step 3: and sequentially synchronizing the remote control instruction and the telemetry alarm data. Because daily remote control instructions and telemetry alert data storage are smaller than telemetry data, synchronization is efficient without requiring segmentation.
Real-time parallel synchronization refers to that a plurality of dynamic data sources can actively perform data synchronization in parallel at a certain time interval in real time, and the specific steps are as follows:
step 1: the dynamic data source is registered. After the system is started, the data source interface is dynamically initialized according to the system configuration file.
Step 2: multiple data sources are synchronized in parallel. The system performs data synchronization of each data source in parallel by starting up the multithreading.
Step 3: the current synchronization position is recorded. After the first synchronization is finished, whether the synchronization is finished is judged again according to the synchronization position, whether data which is missed is detected, and the synchronization position is updated again after the incremental data is synchronized (aiming at the data on the same day).
Step 4: the latest data is cyclically synchronized. If not, executing step 2 (i.e. full data synchronization); otherwise, incremental data synchronization is performed (parallel data synchronization is performed only for the data of the day, and the synchronized data is filtered by the date of the day). If the data query requirement exists in the synchronization process, the synchronization is continued after waiting for the data query to finish. The synchronization process is actively synchronized once every 1 minute.
The whole flow of data synchronization of the cross-database is shown in figure 2, and mainly comprises six key technical points of multi-data source combination, high reliability, real-time synchronization, cache mechanism introduction, parallel execution, full-quantity and increment synchronization and the like.
a) Multiple data source combination:
in the satellite mass production development stage, related data are required to be stored for each satellite individually, and flexible dynamic expansion is required, so that ground test software is required to support the dynamic configuration of expandability when a database is stored. As shown in FIG. 1, the inventive scheme combines static data sources (column-wise) with multiple dynamic data sources (row-wise) for hybrid deployment and supports dynamic scalable updating of dynamic data sources, including the number of dynamic data sources and its own database configuration. Dynamically adding or deleting or modifying data source configuration by modifying system configuration file, then restarting service to take effect,
b) High reliability:
in order to ensure the stability of data synchronization, the system adopts a cluster deployment scheme of haproxy+keepalive-main and standby, and under the condition that a main server of the system is down, a standby server automatically preempts drift IP, so that the service is continuously ensured not to be interrupted, and the system stability can be greatly improved.
c) Real-time synchronization:
because the satellite is in the ground test stage, the test data needs to be queried in real time, and the traditional row-type relational database cannot meet the requirement of the satellite test order data query performance, the scheme can effectively realize the read-write separation of the data and improve the overall usability of the system by introducing a row-type storage and column-type query combined database deployment mode to meet the requirement, and the real-time synchronization of the row-type database and the column-type database becomes a necessary requirement. According to the invention, under the requirement of ensuring the query performance, data is actively synchronized once every 1 minute, so that the real-time query requirement of satellite test is basically met.
d) Introducing a caching mechanism:
a caching mechanism is introduced, considering that a user may perform the same or similar query needs multiple times. When the data is queried, if the cache is not hit, the data is recorded in the cache for storage, and when the next query is performed, the data can be hit from the cache and the query result can be directly returned from the cache to the user. Under the condition that the user inquires the miss cache, the data synchronization can wait for 5 seconds, then whether the inquiry is completed is detected, and after repeated detection, if the inquiry is completed, the data synchronization is continued, and the scheme improves the reliability of the data inquiry and the performance requirement of the data synchronization.
e) And (3) parallel execution:
considering the case where a plurality of satellites are simultaneously tested, it is required that data synchronization can be performed in parallel in order to improve the performance of synchronizing data. Under the condition that the physical resources (mainly CPU cores) of the server meet the requirements, an asynchronous multithreading synchronous mechanism is adopted, the parallel synchronous requirement of satellite testing is met, and the data synchronous performance requirement is greatly improved. The synchronization mechanism can wait for the synchronization position of each satellite to be recorded after all data are synchronized, so that new data can be conveniently synchronized from the recorded position continuously.
f) Full and delta synchronization:
in order to improve the stability of data synchronization, a data synchronization mechanism combining a full quantity mode and an increment mode is adopted. The incremental data synchronization aims at the data on the same day, and can meet the continuous updating synchronization of the data on the same day; for full data synchronization, two situations can be distinguished: the first is that the batch data are synchronized once when the system is started for the first time, and the second is that the data with the left synchronization are checked again for updating again when the system is started for one day. In general, the system meets the synchronous requirement of daily real-time update, also meets the stability requirement of data synchronization, and fully ensures the real-time and reliability requirements of data synchronization.
In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and the division of modules, or units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units, modules, or components may be combined or integrated into another apparatus, or some features may be omitted, or not performed.
The units may or may not be physically separate, and the components shown as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such embodiments, the computer program may be downloaded and installed from a network via a communication portion, and/or installed from a removable medium. The above-described functions defined in the method of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU). The computer readable medium of the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The foregoing is merely illustrative of specific embodiments of the present invention, and the scope of the present invention is not limited thereto, but any changes or substitutions within the technical scope of the present invention should be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. The real-time parallel synchronization method across databases is characterized by comprising the following steps of:
registering a plurality of dynamic data sources;
synchronizing data from a dynamic data source to a static data source, and recording a synchronization position;
acquiring a query instruction;
and according to the query instruction, querying the data from the corresponding synchronous position of the static data source.
2. The method for real-time parallel synchronization across databases according to claim 1, wherein the step of synchronizing data from a dynamic data source to a static data source comprises the steps of:
acquiring a current dynamic data source interface according to the satellite unique mark of the dynamic data source;
the dynamic data source is synchronized to the static data source through the corresponding current dynamic data source interface.
3. The method for real-time parallel synchronization across databases according to claim 2, wherein:
the data of the dynamic data source comprises telemetry data, remote control instructions and telemetry alarm data.
4. A method of real-time parallel synchronization across databases according to claim 3, wherein the dynamic data sources are synchronized to static data sources via respective current dynamic data source interfaces, comprising the steps of:
obtaining a data table according to the data of the dynamic data source and the static data source;
according to the data table, the dynamic data source sequentially synchronizes telemetry data to the static data source through the corresponding current dynamic data source interface;
the dynamic data source synchronizes the remote control instruction and the telemetry alarm data to the static data source through the corresponding current dynamic data source interface.
5. The method of real-time parallel synchronization across databases of claim 4, further comprising the steps of:
in the synchronization process, recording the synchronization state of the data table;
taking the maximum ID position in the data table as a synchronous ending position;
in the data synchronization process, whether the synchronization is completed or not is judged according to the synchronization ending position.
6. The method of real-time parallel synchronization across databases of claim 5, further comprising the steps of:
after the first synchronization of the data is completed, judging whether the synchronization is completed or not according to the synchronization position, and detecting whether the data is missed.
7. The method for real-time parallel synchronization across databases according to claim 6, wherein:
the dynamic data source synchronizes data to the static data source every day and updates the synchronization position.
8. The method of real-time parallel synchronization across databases of claim 7, further comprising the steps of:
and sending the data obtained by inquiring in the static data source to the cache.
9. The method for real-time parallel synchronization across databases according to claim 8, wherein the step of querying data from the corresponding synchronization location of the static data source according to the query command comprises the steps of:
inquiring whether corresponding data exists in the cache according to the satellite unique mark of the inquiring instruction;
if the corresponding data exists in the cache, downloading the data;
if the corresponding data does not exist in the cache, inquiring whether the corresponding data exists in the static data source;
if the corresponding data exists in the static data source, downloading the data;
if the static data source does not have corresponding data, synchronizing the data from the dynamic data source to the static data source every 5S until the corresponding data in the static data source is inquired.
10. A real-time parallel synchronization system across databases for implementing the real-time parallel synchronization method across databases according to any one of claims 1 to 9, comprising:
the registration module is used for registering a plurality of dynamic data sources;
the synchronization module is used for synchronizing data from the dynamic data source to the static data source and recording the synchronization position;
the acquisition module is used for acquiring the query instruction;
and the query module is used for querying data from the corresponding synchronous positions of the static data source according to the query instruction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310818394.0A CN116894071A (en) | 2023-07-05 | 2023-07-05 | Real-time parallel synchronization method and system for cross database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310818394.0A CN116894071A (en) | 2023-07-05 | 2023-07-05 | Real-time parallel synchronization method and system for cross database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116894071A true CN116894071A (en) | 2023-10-17 |
Family
ID=88314320
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310818394.0A Pending CN116894071A (en) | 2023-07-05 | 2023-07-05 | Real-time parallel synchronization method and system for cross database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116894071A (en) |
-
2023
- 2023-07-05 CN CN202310818394.0A patent/CN116894071A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110321387B (en) | Data synchronization method, equipment and terminal equipment | |
| US11789938B1 (en) | Ensuring globally consistent transactions | |
| WO2015154694A1 (en) | System and method for massively parallel processing database | |
| CN103678494A (en) | Method and device for client side and server side data synchronization | |
| US11860847B2 (en) | Efficient production and consumption for data changes in a database under high concurrency | |
| US8510602B2 (en) | Testing a software application used in a database system | |
| CN110321383A (en) | Big data platform method of data synchronization, device, computer equipment and storage medium | |
| US12108311B2 (en) | Vehicular apparatus, and time synchronization method for vehicular apparatus | |
| EP3620932A1 (en) | Method and system for merging data | |
| CN113064919B (en) | Data processing method, data storage system, computer device and storage medium | |
| CN114416868A (en) | Data synchronization method, device, equipment and storage medium | |
| CN112417043B (en) | Data processing system and method | |
| CN114328739A (en) | Data synchronization method, data reading method, data synchronization device, data reading device, electronic equipment, storage medium and product | |
| CN114398334A (en) | Prometheus remote storage method and system based on ZNBase cluster | |
| CN101425078A (en) | Software source code updating method and device | |
| CN113468143A (en) | Data migration method, system, computing device and storage medium | |
| CN116894071A (en) | Real-time parallel synchronization method and system for cross database | |
| CN113626449A (en) | Data storage method, data query method and related equipment | |
| CN116842244A (en) | Search engine data synchronization method, system, device and storage medium | |
| CN113761052A (en) | Database synchronization method and device | |
| CN102413154B (en) | Method of data information synchronization between systems and apparatus thereof | |
| CN108900497B (en) | Data synchronization method and system between heterogeneous systems | |
| CN115113989B (en) | Transaction execution method, device, computing equipment and storage medium | |
| CN109558417B (en) | Data processing method and system | |
| CN117171266B (en) | Data synchronization method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |