CN115510155A - Data dynamic synchronization method and edge user terminal - Google Patents

Data dynamic synchronization method and edge user terminal Download PDF

Info

Publication number
CN115510155A
CN115510155A CN202211157791.XA CN202211157791A CN115510155A CN 115510155 A CN115510155 A CN 115510155A CN 202211157791 A CN202211157791 A CN 202211157791A CN 115510155 A CN115510155 A CN 115510155A
Authority
CN
China
Prior art keywords
data
edge user
request
data request
user terminal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211157791.XA
Other languages
Chinese (zh)
Inventor
陆铭洲
徐海林
张雨晨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
711th Research Institute of CSIC
Original Assignee
711th Research Institute of CSIC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 711th Research Institute of CSIC filed Critical 711th Research Institute of CSIC
Priority to CN202211157791.XA priority Critical patent/CN115510155A/en
Publication of CN115510155A publication Critical patent/CN115510155A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/273—Asynchronous replication or reconciliation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2365—Ensuring data consistency and integrity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • G06F16/2386—Bulk updating operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data dynamic synchronization method and an edge user terminal, comprising the following steps: the method comprises the following steps: step S1: copying and distributing any edge user terminal data request information to other edge user terminals; step S2: comparing the time information of the latest data request of each edge user terminal, and judging whether data loss exists or not; if any edge user terminal has data loss, performing historical data request record synchronization on the edge user terminal; if no data loss exists in the edge user terminal, the data request is executed normally; and step S3: and updating the historical data request record after the data request is successfully executed. The reliability and the integrity of the data are ensured in a global backup rather than distributed storage mode; the whole data synchronization process only needs to configure the IP address of the edge user terminal, so that the configuration information is simplified to the maximum extent, and the operability is improved.

Description

Data dynamic synchronization method and edge user terminal
Technical Field
The invention belongs to the field of comprehensive automation of ships, and particularly relates to a dynamic data synchronization method and an edge user terminal.
Background
In the integrated automation system of the ship, a plurality of edge user terminals need to keep consistent data all the time, and simultaneously, the integrated automation system has the following relevant requirements in the field of the integrated automation of the ship: the reliability is high, the user terminals are independent and do not influence each other, and the usability of software and the integrity of data can be ensured even under the extreme condition that only one user terminal can operate; the flexibility is realized, and various data sources and data types in the ship industrial field can be adapted; ease of operation, a crew member other than a software professional can also modify the configuration to account for possible emergencies. Therefore, a low latency, highly reliable, flexible, easy to operate synchronization method that ensures data integrity is very important and meaningful.
At present, two mainstream data synchronization methods are provided, one is a method based on a database cluster, and the database of all terminals is virtualized into a cluster to integrally manage and store data; the other method is a synchronization method based on master-slave backup, and comprises two modes of database backup and hard disk hardware backup. The method based on the database cluster has the disadvantages that firstly, data in the cluster are actually stored in each machine according to strategy distribution, and disconnection loss of a large number of nodes in the cluster can cause data loss and even unavailability of the whole cluster; and secondly, the crash of the whole cluster caused by operation errors can occur, so that all cluster data is unavailable, and the reliability of the system is reduced. The synchronization method based on master-slave backup has the disadvantages that firstly, a host needs to be appointed, and if data loss or errors occur in the host, errors occur in the whole data; and secondly, the method depends on specific data storage management software, has low flexibility and high configuration and maintenance difficulty, needs relative personnel to have stronger professional skills and is difficult to operate. It can be seen that neither of the above two schemes can compromise high reliability, flexibility, ease of operation, and data integrity.
Therefore, the prior art has defects and needs to be improved.
Disclosure of Invention
The invention aims to provide a data dynamic synchronization method and an edge user terminal, and aims to solve the problem that the existing data synchronization method cannot simultaneously give consideration to high reliability, flexibility, easiness in operation and data integrity.
The invention provides a data dynamic synchronization method, which comprises the following steps: step S1: copying and distributing any edge user terminal data request information to other edge user terminals; step S2: comparing the time information of the latest data request of each edge user terminal, and judging whether data loss exists or not; if any edge user terminal has data loss, performing historical data request record synchronization on the edge user terminal; if no data loss exists in the edge user terminal, the data request is executed normally; and step S3: and updating the historical data request record after the data request is successfully executed.
Optionally, in some embodiments of the present invention, in step S1, any edge user terminal data request information is copied and distributed to the remaining edge user terminals through the intermediate proxy server; and configuring configuration information of all the edge user terminals in the intermediate proxy server.
Optionally, in some embodiments of the invention, the intermediate proxy server comprises a Nginx intermediate proxy server.
Optionally, in some embodiments of the present invention, in the step S2, a latest data request time of the edge user terminal to be data-synchronized is queried in a cache of the edge user terminal, and a request is sent to query a latest data request time of caches of the other edge user terminals, and then the latest data request time of the edge user terminal to be data-synchronized is compared with the latest data request time of the other edge user terminals in a time sequence; if the latest data request time of the edge user terminal to be subjected to data synchronization is before, judging that data is missing, and performing historical data request record synchronization on the edge user terminal; and if the latest data request time of the edge user terminal to be subjected to data synchronization is later, judging that no data is missing, and normally executing the data request.
Optionally, in some embodiments of the present invention, when performing history data request record synchronization on an edge user terminal to be synchronized with data, the method includes the following steps: step S21: sending a request to acquire all historical data request records of the rest edge user terminals after the latest data request time of the edge user terminal to be subjected to data synchronization; step S22: according to the request time information of the historical data request records, integrating the historical data request records of all the edge user terminals into a union set which is ordered according to time and is not repeated; step S23: sequentially sending simulation requests to the edge user terminals to be subjected to data synchronization according to the union of the historical data request records of the edge user terminals; if the simulation request fails, the simulation request is sent again after the preset delay time; and if the simulation request is successful, normally executing the data request.
Optionally, in some embodiments of the present invention, the emulation request sends an emulation HTTP request using RestTemplate.
Optionally, in some embodiments of the present invention, in step S3, after the data request is successfully executed, the data request is added to the batch data processing program, and the history data request record is updated.
Optionally, in some embodiments of the present invention, the historical data request record includes: request address, request header information, request body, request time and the number of the edge user terminal.
Optionally, in some embodiments of the invention, the historical data request record is maintained in a separate mysql database of the InNODB engine.
Correspondingly, the invention also provides an edge user terminal, comprising: the operation interface is used for sending data request information; the intermediate proxy server is used for copying and distributing the data request information to other edge user terminals; a history check program for integrating the history data request records of all the edge user terminals; a batch data processing program for updating the historical data request record; and the method for performing the dynamic data synchronization among a plurality of edge user terminals.
In summary, the invention provides a data dynamic synchronization method and an edge user terminal, which convert common business stock data into incremental record data by means of independently storing data request records and comparing latest request time, thereby avoiding query and comparison of global data, reducing database pressure caused by synchronous data, and simultaneously completely decoupling from a specific business database, thereby improving flexibility; by replacing a timing asynchronous inspection strategy with a strategy of triggering synchronization by a request, the operation on the service data is reduced to the minimum extent under the condition of low delay, and the performance of the service data is not influenced; the reliability and the integrity of the data are ensured in a global backup rather than distributed storage mode; the whole synchronization process only needs to configure the IP address of the edge user terminal, so that the configuration information is simplified to the maximum extent, and the operability is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a schematic flow chart of a history check and data synchronization method of a dynamic data synchronization method according to an embodiment of the present invention;
fig. 2 is a schematic structural relationship diagram of an edge user terminal in a dynamic data synchronization process in the embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or to imply that the number of technical features indicated is significant. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless otherwise specified.
Referring to fig. 1 and fig. 2, the present invention provides a data dynamic synchronization method, which includes the following steps:
step S1: and copying and distributing the data request information of any edge user terminal to the rest edge user terminals.
Specifically, any edge user terminal receives new data request information, the edge user terminal is a source edge user terminal of the data request information, and the source edge user terminal copies the data request information through the intermediate proxy server and distributes the data request information to the rest edge user terminals. Wherein the intermediate proxy server is preferably a Nginx intermediate proxy server. Nginx is a high-performance HTTP and reverse proxy web server, which has less memory and strong concurrency capability, provides a traffic mirroring function, and can collect all requests, copy and distribute the requests to other environments. And the ip addresses of all edge user terminals are configured in Nginx, and the whole historical data request record synchronization step only needs to configure the ip addresses of the edge user terminals, so that the configuration information is simplified to the maximum extent, and the operability is improved.
Step S2: comparing the time information of the latest data request of each edge user terminal, and judging whether data loss exists or not; if any edge user terminal has data loss, performing historical data request record synchronization on the edge user terminal; and if no data loss exists in the edge user terminal, normally executing the data request. It can be understood that, usually, in the same time, not all the edge ues are source edge ues, and at this time, the data request information received in each edge ue is not the same, this step can determine the edge ue lacking the data request information, and synchronize the history data request record with the edge ue without data loss; and the edge user terminal without data loss directly skips the step of synchronization of historical data request records and executes the data request.
As shown in fig. 2, in particular, in the step S2, at the edge user terminal to be data synchronizedThe cache of the edge user terminal inquires the latest data request time of the edge user terminal, sends a request to inquire the latest data request time of the caches of the other edge user terminals, and then compares the latest data request time of the edge user terminal to be subjected to data synchronization with the latest data request time of the other edge user terminals in time sequence; if the latest data request time of the edge user terminal to be subjected to data synchronization is before, judging that data is missing, and performing historical data request record synchronization on the edge user terminal; and if the latest data request time of the edge user terminal to be subjected to data synchronization is later, judging that no data is missing, and normally executing the data request. That is, the time t of the last data request is inquired from the cache of the edge user terminal to be data synchronized 0 Meanwhile, sending data request to inquire the last request time [ t ] in the caches of other edge user terminals 1 ，t 2 ，...，t n ]Judging the last request time t of the local computer 0 Whether it is the last time requested by all edge terminals t 0 ，t 1 ，t 2 ，...，t n ]Maximum value of (1), if t 0 If the maximum value is, the data request is processed normally, if t is 0 Is a non-maximum value (assuming t n Maximum value), the historical data request record synchronization is carried out on the edge user terminal to be synchronized with the data.
Further, when the edge user terminal to be data synchronized performs the historical data request record synchronization, the method comprises the following steps:
step S21: sending a request to acquire all historical data request records t of the rest edge user terminals after the latest data request time of the edge user terminal to be subjected to data synchronization n ；
Step S22: according to the request time information of the historical data request records, integrating the historical data request records of all the edge user terminals into a union set [ t ] which is ordered according to time and does not repeat 0 ，t 1 ，t 2 ，...，t n ]；
Step S23: sequentially sending simulation requests to the edge user terminals to be subjected to data synchronization according to the union of the historical data request records of the edge user terminals; if the simulation request fails, the simulation request is sent again after the preset delay time; and if the simulation request is successful, normally executing the data request. And sending the simulation HTTP request by adopting RestTemplate. Preferably, if the data request fails or times out, the data request is retried after being delayed for 1 second, the retry is performed for three times in total, if the data request still fails after three times, the step of historical data request record synchronization is directly skipped, and the record of the data request failure is correspondingly stored, so that the repair synchronization can be conveniently performed manually in the follow-up process. By replacing the timing asynchronous inspection strategy with the strategy of triggering synchronization by request, the operation on the service data is reduced to the minimum extent under the condition of low delay, and the performance of the service data is not influenced.
And step S3: and after the data request is successfully executed, updating the historical data request record.
Specifically, after the data request is successfully executed, each edge user terminal adds the data request to the batch data processing program, and updates the history data request record through the batch data processing program, so as to synchronize the data of each edge user terminal.
Preferably, the historical data request record comprises: the request address, the request header information, the request body, the request time, and the number of the edge user terminal, etc., but are not limited thereto.
Preferably, the historical data request record is stored in an independent mysql database of the InNODB engine, that is, the historical data request record is different from the storage position of the main service database, and common service stock data is converted into incremental record data in a mode of independently storing the request record and comparing the latest request time, so that query and comparison of global data are avoided, database pressure caused by synchronous data is reduced, and meanwhile, the historical data request record is completely decoupled from a specific service database, and flexibility is improved. The reliability and the integrity of the data are ensured by a global backup rather than a distributed storage mode.
The invention also provides an edge user terminal, which is used for realizing the data dynamic synchronization method and comprises the following steps: the operation interface is used for sending data request information; the intermediate proxy server is used for copying and distributing the data request information to other edge user terminals; a history check program for integrating the history data request records of all the edge user terminals; a batch data processing program for updating the historical data request record; and the method for performing the dynamic data synchronization among a plurality of edge user terminals.
Preferably, the history check program is a HandlerInterreceptor interceptor in the main program. The historical verification program adopts rpc remote calling framework when verifying related requests, does not pass through intermediate proxy servers of other edge terminal equipment, adopts multi-thread asynchronous calling when calling, and the main thread executes judgment operation after all the sub-threads are called. And if the query calling fails or is overtime, directly skipping the historical data synchronization flow.
In summary, the invention provides a data dynamic synchronization method and an edge user terminal, which provide a way of storing data request records separately and comparing the latest request time, convert common business stock data into incremental record data, avoid query and comparison of global data, reduce database pressure caused by synchronous data, and simultaneously completely decouple from a specific business database, thereby improving flexibility; by replacing a timing asynchronous inspection strategy with a strategy of triggering synchronization by a request, the operation on the service data is reduced to the minimum extent under the condition of low delay, and the performance of the service data is not influenced; the reliability and the integrity of the data are ensured in a global backup rather than distributed storage mode; the whole synchronization process only needs to configure the IP address of the edge user terminal, so that the configuration information is simplified to the maximum extent, and the operability is improved.
Embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The technical solutions provided by the embodiments of the present invention are described in detail above, and the principles and embodiments of the present invention are explained in the present document by applying specific examples, and the descriptions of the above embodiments are only used to help understanding the method and the core idea of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed, and in summary, the content of the present specification should not be construed as limiting the present invention.

Claims (10)

1. A dynamic data synchronization method is characterized by comprising the following steps:
step S1: copying and distributing any edge user terminal data request information to other edge user terminals;
step S2: comparing the time information of the latest data request of each edge user terminal, and judging whether data loss exists or not; if any edge user terminal has data loss, performing historical data request record synchronization on the edge user terminal; if no data loss exists in the edge user terminal, the data request is executed normally;
and step S3: and after the data request is successfully executed, updating the historical data request record.
2. The data dynamic synchronization method according to claim 1, wherein in the step S1, any edge user terminal data request information is copied and distributed to the rest of the edge user terminals through the intermediate proxy server; wherein
And the intermediate proxy server is configured with configuration information of all the edge user terminals.
3. The method for dynamically synchronizing data according to claim 2, wherein the intermediate proxy server comprises a Nginx intermediate proxy server.
4. The dynamic data synchronization method according to claim 1, wherein in step S2, the last data request time of the edge ue to be data synchronized is queried in its cache, and a request is sent to query the last data request times of the other caches of the edge ues, and then the last data request time of the edge ue to be data synchronized is compared with the last data request times of the other edge ues in time sequence; if the latest data request time of the edge user terminal to be subjected to data synchronization is before, judging that data is missing, and performing historical data request record synchronization on the edge user terminal; and if the latest data request time of the edge user terminal to be subjected to data synchronization is later, judging that no data is missing, and normally executing the data request.
5. The dynamic data synchronization method of claim 4, wherein when the edge user terminals to be synchronized with data are synchronized with historical data request records, the method comprises the following steps:
step S21: sending a request to acquire all historical data request records of the rest edge user terminals after the latest data request time of the edge user terminal to be subjected to data synchronization;
step S22: according to the request time information of the historical data request records, integrating the historical data request records of all the edge user terminals into a union set which is ordered according to time and is not repeated;
step S23: sequentially sending simulation requests to the edge user terminals to be subjected to data synchronization according to the union of the historical data request records of the edge user terminals; if the simulation request fails, the simulation request is sent again after the preset delay time; and if the simulation request is successful, normally executing the data request.
6. The method for dynamic synchronization of data of claim 5, wherein the emulation request sends an emulation HTTP request using RestTemplate.
7. The method for dynamically synchronizing data according to claim 1, wherein in step S3, after the data request is successfully executed, the data request is added to the batch data processing program, and the history data request record is updated.
8. The method for dynamic synchronization of data of claim 1, wherein the historical data request record comprises: request address, request header information, request body, request time and the number of the edge user terminal.
9. The method for dynamic synchronization of data according to claim 1, wherein the historical data request records are maintained in a separate mysql database of the InNODB engine.
10. An edge user terminal, comprising:
the operation interface is used for sending data request information;
the intermediate proxy server is used for copying and distributing the data request information to other edge user terminals;
a history check program for integrating the history data request records of all the edge user terminals;
a batch data processing program for updating the historical data request record;
a method of performing dynamic synchronization of data according to any one of claims 1 to 9 between a plurality of said edge user terminals.
CN202211157791.XA 2022-09-22 2022-09-22 Data dynamic synchronization method and edge user terminal Pending CN115510155A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211157791.XA CN115510155A (en) 2022-09-22 2022-09-22 Data dynamic synchronization method and edge user terminal

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211157791.XA CN115510155A (en) 2022-09-22 2022-09-22 Data dynamic synchronization method and edge user terminal

Publications (1)

Publication Number Publication Date
CN115510155A true CN115510155A (en) 2022-12-23

Family

ID=84505343

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211157791.XA Pending CN115510155A (en) 2022-09-22 2022-09-22 Data dynamic synchronization method and edge user terminal

Country Status (1)

Country Link
CN (1) CN115510155A (en)

Similar Documents

Publication Publication Date Title
US7668874B2 (en) Disaster recovery processing method and apparatus and storage unit for the same
US7565572B2 (en) Method for rolling back from snapshot with log
CN102265277B (en) Operation method and device for data memory system
US7779295B1 (en) Method and apparatus for creating and using persistent images of distributed shared memory segments and in-memory checkpoints
US8346719B2 (en) Multi-node replication systems, devices and methods
US7299378B2 (en) Geographically distributed clusters
JP4461147B2 (en) Cluster database using remote data mirroring
US8572037B2 (en) Database server, replication server and method for replicating data of a database server by at least one replication server
CN110990432B (en) Device and method for synchronizing distributed cache clusters across machine room
US10831741B2 (en) Log-shipping data replication with early log record fetching
US20190129976A1 (en) Apparatus for controlling synchronization of metadata on network and method for the same
US8028192B1 (en) Method and system for rapid failback of a computer system in a disaster recovery environment
EP2434729A2 (en) Method for providing access to data items from a distributed storage system
CN113239013B (en) Distributed system and storage medium
CN101079896A (en) A multi-availability mechanism coexistence framework of concurrent storage system
JP5292351B2 (en) Message queue management system, lock server, message queue management method, and message queue management program
US20090063486A1 (en) Data replication using a shared resource
CN113254275A (en) MySQL high-availability architecture method based on distributed block device
US20100274758A1 (en) Data processing method, computer, and data processing program
JP2004348701A (en) Control method for making data dual between computer systems
CN115185787A (en) Method and device for processing transaction log
CN115292408A (en) Master-slave synchronization method, device, equipment and medium for MySQL database
WO2021082925A1 (en) Transaction processing method and apparatus
CN115510155A (en) Data dynamic synchronization method and edge user terminal
JP2002358222A (en) Data dual system and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination