CN113901147A - Data synchronization method and system - Google Patents
Data synchronization method and system Download PDFInfo
- Publication number
- CN113901147A CN113901147A CN202111226678.8A CN202111226678A CN113901147A CN 113901147 A CN113901147 A CN 113901147A CN 202111226678 A CN202111226678 A CN 202111226678A CN 113901147 A CN113901147 A CN 113901147A
- Authority
- CN
- China
- Prior art keywords
- data
- buffer area
- data stream
- synchronization
- setting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to the field of data synchronization, in particular to a data synchronization method and a data synchronization system, which greatly improve the efficiency of data transmission and the stability of data synchronization under the condition of overlarge synchronous data amount of a database. The data synchronization method of the invention comprises the following steps: setting query parameters, querying corresponding data in a database through the query parameters, and sorting the queried corresponding data into data in a JSON format; setting a data stream sending buffer area, adding data in a JSON format to the data stream buffer area, and waiting for initialization; after initialization, the data stream sending buffer zone transmits data to a receiving party in an interface transmission mode; setting a data stream receiving buffer area, and putting the received data into the data stream receiving buffer area; and converting the data of the data stream receiving buffer area into a corresponding format according to a synchronous database updating method and then updating the data into the synchronous database. The invention is suitable for data synchronization when the data volume is overlarge.
Description
Technical Field
The invention relates to the field of data synchronization, in particular to a data synchronization method and a data synchronization system.
Background
During client development, a developer often deploys a plurality of databases in order to conveniently test data without influencing the use of users, and the data of the databases needs to be synchronized regularly or manually by the developer. When data synchronization is carried out, a network protocol such as http is required to be used for transmission, but the protocol generally has certain transmission limitation, and the overlarge data volume can exceed the transmission limitation of the protocol when the database is synchronized, so that transmission failure is caused, the data transmission efficiency is poor, and the data synchronization stability is poor.
Disclosure of Invention
The invention aims to provide a data synchronization method and a data synchronization system, which greatly improve the efficiency of data transmission and the stability of data synchronization under the condition of overlarge database synchronization data volume.
The invention adopts the following technical scheme to realize the purpose, and the data synchronization method comprises the following steps:
step 1, setting query parameters, and querying corresponding data in a database through the query parameters;
step 2, the inquired corresponding data is arranged into data in a JSON format;
step 3, setting a data stream sending buffer area, adding data in a JSON format into the data stream buffer area, and waiting for initialization;
step 4, after initialization, the data stream sending buffer area transmits the data to a receiving party in an interface transmission mode;
step 5, setting a data stream receiving buffer area, and putting the received data into the data stream receiving buffer area;
and 6, converting the data of the data stream receiving buffer area into a corresponding format according to a synchronous database updating method and then updating the data into the synchronous database.
Further, in step 1, the number of the query parameters is multiple.
Further, in step 4, the interface is an http interface.
Further, the method for transmitting data through the http interface specifically includes:
step 301, creating an http policy object, and setting a request parameter and a request address through an http policy object instance;
step 302, forwarding the request parameter and the request address to a specified client address through the execution request;
step 303, the client acquires the request parameters and the specific data in the request address by the corresponding method in the example;
and step 304, the client analyzes and acquires data according to the request parameters and the specific data in the request address.
The data synchronization system is used for executing the data synchronization method, and comprises the following steps:
the data acquisition module is used for setting query parameters and querying corresponding data in the database through the query parameters;
the data sending processing module is used for sorting the inquired corresponding data into data in a JSON format, setting a data stream sending buffer area, adding the data in the JSON format into the data stream buffer area, and waiting for data sending;
the initialization module is used for transmitting data to a receiver by a data stream sending buffer area in an interface transmission mode after initialization;
the data receiving and processing module is used for setting a data stream receiving buffer area and putting the received data into the data stream receiving buffer area;
and the synchronization module is used for converting the data of the data stream receiving buffer area into a corresponding format according to the synchronization database updating method and then updating the data into the synchronization database.
The invention sets a data stream sending buffer area, adds the data to be sent to the data stream buffer area, and sends the data by the buffer area, and sets a data stream receiving buffer area, and when receiving, the buffer area also receives the data; and finally, the data in the data stream receiving buffer area is converted into a corresponding format according to a synchronous database updating method and then updated into the synchronous database, so that the problem that the transmission data volume exceeds the transmission protocol limit when the data in the database is synchronized is effectively solved, a user can select more required data for synchronization at one time, the synchronization efficiency is improved, and the stability of data synchronization is also improved.
Drawings
FIG. 1 is a flow chart of a method for data synchronization according to the present invention.
FIG. 2 is a flow chart of the present invention for transferring data using an http interface.
Detailed Description
The data synchronization method of the invention comprises the following steps:
step 1, setting query parameters, and querying corresponding data in a database through the query parameters;
step 2, the inquired corresponding data is arranged into data in a JSON format;
step 3, setting a data stream sending buffer area, adding data in a JSON format into the data stream buffer area, and waiting for initialization;
step 4, after initialization, the data stream sending buffer area transmits the data to a receiving party in an interface transmission mode;
step 5, setting a data stream receiving buffer area, and putting the received data into the data stream receiving buffer area;
and 6, converting the data of the data stream receiving buffer area into a corresponding format according to a synchronous database updating method and then updating the data into the synchronous database.
In step 1, the number of the query parameters is multiple.
In step 4, the interface is an http interface.
The method for transmitting data through the http interface specifically includes:
step 301, creating an http policy object, and setting a request parameter and a request address through an http policy object instance;
step 302, forwarding the request parameter and the request address to a specified client address through the execution request;
step 303, the client acquires the request parameters and the specific data in the request address by the corresponding method in the example;
and step 304, the client analyzes and acquires data according to the request parameters and the specific data in the request address.
The data synchronization system is used for executing the data synchronization method, and comprises the following steps:
the data acquisition module is used for setting query parameters and querying corresponding data in the database through the query parameters;
the data sending processing module is used for sorting the inquired corresponding data into data in a JSON format, setting a data stream sending buffer area, adding the data in the JSON format into the data stream buffer area, and waiting for data sending;
the initialization module is used for transmitting data to a receiver by a data stream sending buffer area in an interface transmission mode after initialization;
the data receiving and processing module is used for setting a data stream receiving buffer area and putting the received data into the data stream receiving buffer area;
and the synchronization module is used for converting the data of the data stream receiving buffer area into a corresponding format according to the synchronization database updating method and then updating the data into the synchronization database.
FIG. 1 is a flow chart of a method for data synchronization according to the present invention, as shown in the figure: firstly, inquiring data according to selected parameters, then converting the data into a JSON format, and storing the data into an output buffer area through an input buffer stream; after receiving the data, storing the data into an input buffer area, and transmitting the data between the output buffer area and the input buffer area through an http protocol; then the data input into the buffer area is input in the form of output buffer stream, and the data is converted into the required format, thus completing the synchronous function of the database.
FIG. 2 is a flow chart of the present invention for transmitting data using an http interface, as shown in the figure:
firstly, an http policy object is created, then a request parameter and a request address are set through an http policy object instance, the request parameter and the request address are forwarded to a specified client address through an execution request, the client obtains specific data in the set parameters through a corresponding method in the instance, and the client analyzes and obtains the data according to the request parameter and the specific data in the request address, thereby realizing data synchronization.
The method and the system use IO stream as a data transmission mode, realize the function of data synchronization between databases, and solve the problem that the data volume exceeds the limit of http transmission size when the databases are synchronized; the specific transmission mode realizes the mode of completing the access to other interfaces in the interfaces by creating an HttpClient object and transmitting information as a parameter to a received interface address, thereby solving the problem of inconsistent data among different environments.
In summary, the invention effectively solves the problem that the transmission data volume exceeds the transmission protocol limit when the database synchronizes the data, so that the user can select more required data for synchronization at one time, thereby improving the synchronization efficiency and the stability of data synchronization.
Claims (5)
1. A data synchronization method, comprising:
step 1, setting query parameters, and querying corresponding data in a database through the query parameters;
step 2, the inquired corresponding data is arranged into data in a JSON format;
step 3, setting a data stream sending buffer area, adding data in a JSON format into the data stream buffer area, and waiting for initialization;
step 4, after initialization, the data stream sending buffer area transmits the data to a receiving party in an interface transmission mode;
step 5, setting a data stream receiving buffer area, and putting the received data into the data stream receiving buffer area;
and 6, converting the data of the data stream receiving buffer area into a corresponding format according to a synchronous database updating method and then updating the data into the synchronous database.
2. The data synchronization method according to claim 1, wherein in step 1, the number of the query parameters is multiple.
3. The data synchronization method according to claim 1, wherein in step 4, the interface is an http interface.
4. The data synchronization method according to claim 3, wherein the manner of transmitting data via the http interface specifically comprises:
step 301, creating an http policy object, and setting a request parameter and a request address through an http policy object instance;
step 302, forwarding the request parameter and the request address to a specified client address through the execution request;
step 303, the client acquires the request parameters and the specific data in the request address by the corresponding method in the example;
and step 304, the client analyzes and acquires data according to the request parameters and the specific data in the request address.
5. A data synchronization system for performing the data synchronization method of any one of claims 1 to 4, comprising:
the data acquisition module is used for setting query parameters and querying corresponding data in the database through the query parameters;
the data sending processing module is used for sorting the inquired corresponding data into data in a JSON format, setting a data stream sending buffer area, adding the data in the JSON format into the data stream buffer area, and waiting for data sending;
the initialization module is used for transmitting data to a receiver by a data stream sending buffer area in an interface transmission mode after initialization;
the data receiving and processing module is used for setting a data stream receiving buffer area and putting the received data into the data stream receiving buffer area;
and the synchronization module is used for converting the data of the data stream receiving buffer area into a corresponding format according to the synchronization database updating method and then updating the data into the synchronization database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111226678.8A CN113901147A (en) | 2021-10-21 | 2021-10-21 | Data synchronization method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111226678.8A CN113901147A (en) | 2021-10-21 | 2021-10-21 | Data synchronization method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113901147A true CN113901147A (en) | 2022-01-07 |
Family
ID=79025943
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111226678.8A Pending CN113901147A (en) | 2021-10-21 | 2021-10-21 | Data synchronization method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113901147A (en) |
-
2021
- 2021-10-21 CN CN202111226678.8A patent/CN113901147A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101902473B (en) | Method for synchronously updating data based on grid GIS (Geographic Information System) | |
| CN110661891B (en) | Cross-network file instant transmission method and system based on unidirectional network | |
| CN111046100B (en) | Method and system for synchronizing relational database to non-relational database | |
| CN105450712B (en) | A kind of data transmission method and device | |
| CN103139299B (en) | Cloud service dispatching method between cloudy and system | |
| CN101686145A (en) | Equipment capable of upgrading, upgrading method and system | |
| WO2018176244A1 (en) | Data collection method, device, and dynamic environment network management system | |
| CN113286002B (en) | Power data transmission method based on multi-priority queue | |
| CN103929377A (en) | Wired network and wireless network combined dispatching method and system and related devices | |
| CN110740195B (en) | Distributed system data synchronization method and system based on message engine | |
| CN105827671A (en) | System platform characterized by distributed use and centralized management and portal server | |
| CN101631148A (en) | Communication method for allocating dynamic addresses in serial communication protocols | |
| CN110971533B (en) | Method, server apparatus, client apparatus, and medium for data communication | |
| CN108197263A (en) | Method of data synchronization | |
| CN112559472B (en) | File transmission method, device, computer equipment and storage medium | |
| CN110674221B (en) | Spatial data synchronization method, terminal and computer readable storage medium | |
| CN113901147A (en) | Data synchronization method and system | |
| CN114490100A (en) | Message queue telemetry transmission load balancing method and device and server | |
| CN107786661B (en) | Information synchronization method | |
| CN101335610B (en) | ARP synchronization method in high-side Ethernet network switch | |
| CN101072172A (en) | Method and device for realizing network data high-efficiency transfer | |
| CN113242313B (en) | Data synchronization method, system, device, server and storage medium | |
| EP2291952A2 (en) | Adapter for synchronizing data over different networks | |
| CN111614612B (en) | Communication protocol implementation method, device, network management server and storage medium | |
| CN202385300U (en) | Data transmission system of internet of things (IOT) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |