CN114756628B - High-multiplexing extensible data and file synchronization method - Google Patents
High-multiplexing extensible data and file synchronization method Download PDFInfo
- Publication number
- CN114756628B CN114756628B CN202210676804.8A CN202210676804A CN114756628B CN 114756628 B CN114756628 B CN 114756628B CN 202210676804 A CN202210676804 A CN 202210676804A CN 114756628 B CN114756628 B CN 114756628B
- Authority
- CN
- China
- Prior art keywords
- data
- synchronization
- information
- database
- servers
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
A method for synchronizing high-multiplexing extensible data and files takes a server or a PC as a carrier and comprises the following steps: releasing synchronous information, namely releasing data information needing to be synchronized, wherein each server is used as a synchronous source to release data; acquiring synchronous information, and acquiring the synchronous information from other servers; deleting synchronous information, periodically inquiring whether the call information of the data synchronous release table of the local database is acquired by other servers, if so, deleting data synchronous information records, and if not, waiting for the next period for inquiry; through the three steps, each server issues information, simultaneously acquires information from other servers, and provides data and file synchronization for other software and applications by taking the database as a synchronization medium. The invention realizes mutual synchronization among a plurality of servers, and realizes file synchronization and database synchronization among a plurality of servers.
Description
Technical Field
The invention relates to the technical field of database and file synchronization of multiple servers, in particular to a high-multiplexing extensible data and file synchronization method.
Background
The traditional data synchronization method in the computer system mainly adopts a synchronization mechanism of database software. For example, SQL Server, which uses a publish-subscribe method to achieve database synchronization, publishes messages by a main Server, subscribes by a backup Server, publishes messages when data of the main Server changes, the backup Server reads messages to perform synchronous update, and the delay of the intermediate process is short.
Problems with this approach are:
(1) When the amount of data to be synchronized is large, the data synchronization method causes the server to be jammed, and normal operation of various programs and services on the server is affected.
(2) A large number of updates or insertions to the publish database may occur with errors or inconsistencies in the data from the subscription servers.
(3) When the number of devices needing synchronization increases, the number of newly-added devices and the number of devices in the original synchronization system need to increase the number of issuing and subscribing transactions, and the deployment is tedious.
(4) Scalability is poor, especially when data synchronization is associated with other services, such as service requirements that require data synchronization to be associated with file synchronization, and the publish-subscribe method of the database cannot be associated with file synchronization.
The above is a traditional database synchronization method, while the traditional file synchronization method generally adopts tools such as rsync or synchronous disk, and if the data volume is small or the requirement on timeliness is not so high, the requirement can be basically met. However, when the amount of data is large, the speed, reliability, stability and timeliness of these conventional synchronization tools are not always sufficient to meet the business requirements.
Taking patent "a data synchronization method, system and storage medium" as an example, publication No. CN110377602a is a traditional data synchronization method, and it also takes a database as a medium, and triggers generation of trigger information and sends the trigger information to a corresponding database table when monitoring a synchronous data sending event operation of a master database. The data synchronization method is suitable for a simple data synchronization scene, a main server is used as a data source, expansibility and high availability are poor, and a synchronization flow is stopped when the main server fails.
In the patent of "a distributed database data synchronization method", publication No. CN113918656B, a central server is used as a synchronization center to receive and issue synchronization information, and this synchronization method requires high requirements on stability and high availability of the central server, and has no explicit processing method for a large amount of data synchronization requirements, which may cause a server jam in an abnormal situation.
Disclosure of Invention
In view of the problems of unsmooth synchronization, complicated synchronization steps of a plurality of servers and poor expansibility in the prior art, the invention provides a method for synchronizing high-multiplexing extensible data and files.
The technical scheme adopted by the invention is as follows: a method for synchronizing high-multiplexing extensible data and files takes a server or PC equipment as a carrier and an operating system as windows or linux, realizes the synchronization of the data and the files among a plurality of equipment, and comprises the following specific steps:
firstly, releasing synchronous information;
data information needing to be synchronized, including time and data content, is published, each server is used as a synchronization source to publish the data, and the publishing method is that the data information is written into a database table, and the type of the database supports a relational database, a non-relational database and a key value database;
secondly, acquiring synchronous information;
acquiring synchronous information from other servers, periodically inquiring whether synchronous data are issued on the other servers, if so, acquiring and updating the synchronous information to a local database, and if not, waiting for the next period to inquire;
thirdly, deleting the synchronous information;
after the data information needing to be synchronized is issued, periodically inquiring whether the data content of a data synchronization issuing table of the local database is acquired by other servers, if so, deleting the data synchronization information record, and if not, waiting for the next period for inquiring;
through the three steps, each server can be used as a synchronization source to release information and simultaneously acquire information from other servers, the synchronization information released by each server is deleted after the synchronization information is acquired, and the database is used as a synchronization medium to provide synchronization of data and files for other servers.
The data content in the first step is that when the data content is applied to different systems, the data content is self-defined according to system requirements, and when the data content is applied to a recording system, the data content comprises a recording initiator, a recording receiver, a recording number, a recording duration and a recording storage path; when the method is applied to a network management system, the data content comprises an alarm type, an alarm level, an alarm source and alarm content.
The method for writing data information into a database table described in the first step is as follows:
the sentence patterns used are different according to different types of the database;
the non-relational database MySQL uses the sentence form as follows:
INSERT INTOs table name (field 1, field 2.. Field n) VALUES (value 1, value 2.. Value n);
the relational database SQLServer uses a sentence pattern of:
update table name set column name = value;
the key value database MongoDB uses sentence patterns as follows:
db. sets the name insertOne ({ key name: value }).
The method for periodically inquiring whether synchronous data are published on other servers in the second step is as follows
The query method uses different sentence patterns according to different types of databases:
the non-relational database MySQL uses the sentence form as follows:
selecting distinting from 'table name' where 'restriction condition' group by 'group, displaying the number of strips' according to 'having' filtering condition 'order by limit';
the relational database SQLServer uses a sentence pattern of:
select from table name where column name = 'value';
the key value database MongoDB uses sentence patterns as follows:
db. sets the name find ({ conditional document });
after the data is inquired, the data is inserted into a local server database, the inquired data is firstly put into a local cache in the inquiry insertion process, then the data is taken out from the local cache and inserted into the local database, in order to more efficiently and more flexibly adjust the synchronization speed according to the data volume, a double-level cache mechanism is introduced and the synchronization speed is changed according to the data volume needing to be synchronized, so that the flexible synchronization process is realized, after the data is acquired, a file synchronization module is informed, the file synchronization is performed according to the value of the data, and the file is acquired from a synchronization source by using an FTP protocol.
The method for deleting the data synchronization information record in the third step is as follows:
due to different database types, the operation statements are different:
a non-relational database MySQL, which uses a sentence pattern of MySQL;
DELETE FROM < table name > [ WHERE clause ] [ ORDER BY clause ] [ LIMIT clause ]
The relational database SQLServer uses the sentence pattern as follows:
delete from table name where column name = 'value';
the key value database MongoDB uses sentence patterns as follows:
db. sets the name remove ({ column name: value });
the synchronization information is deleted from the database table using a deletion schema.
The method for introducing the double-stage cache mechanism is that the synchronization speed is changed according to the size of the data volume to be synchronized, and the flexible synchronization process is realized by the following steps:
after data synchronization information is obtained, the data synchronization information is put into a level 1 cache, then the data is put into a level 2 cache from the level 1 cache, the data synchronization information is obtained from a level 2 cache area and is written into a database, after the data synchronization information is successfully written, the data synchronization information is deleted from the level 1 cache area and the level 2 cache area, a double-level cache mechanism is adopted, when the data quantity needing to be synchronized is too large, the synchronization speed is controllable, and when the data quantity needing to be synchronized is inquired to exceed a threshold value, the period of inquiring and obtaining the data synchronization is adjusted.
The invention has the beneficial effects that: the method and the system realize the correlation of data and file synchronization, solve the problems of unsmooth synchronization, complicated synchronization steps of a plurality of servers and poor expansibility in the traditional data synchronization and file synchronization, have controllable and stable synchronization speed, are easy to expand, are suitable for a plurality of scenes needing data and file synchronization, and have the characteristics of high reusability and expandability.
The multiple servers (> = 2) are mutually synchronous sources, mutual synchronization among the multiple servers can be realized, file synchronization and database synchronization among the multiple servers can be realized, and the method can be applied to multiple operating systems such as windows and linux.
Drawings
FIG. 1 is a flow chart of data synchronization information distribution according to the present invention;
FIG. 2 is a process flow of the exception handling of the data synchronization information issue according to the present invention;
FIG. 3 is a flow of data synchronization information acquisition into level 1 cache according to the present invention;
FIG. 4 is a flow of data synchronization information acquisition into level 2 cache according to the present invention;
FIG. 5 is a flow chart of deleting data synchronization information according to the present invention;
FIG. 6 is a block diagram of the interface class diagram design of the present invention extensible to support multiple databases;
FIG. 7 is a block diagram of an external connection for synchronizing data files in a recording system according to the present invention;
FIG. 8 is a block diagram of the internal components of the recording system for synchronizing data files according to the present invention.
Detailed Description
As shown in fig. 1, the process of issuing data synchronization information: the software is in an idle state by default, periodically judges whether the software accords with the condition of releasing the synchronous information, and if the software accords with the condition, the software uses a database to release the data synchronous information; if not, continuing to periodically judge whether the self meets the condition of releasing the synchronous information; after the data synchronization information is issued, judging whether the data synchronization information is successfully issued, if the data synchronization information is successfully issued, returning to a default idle state, and waiting for the next period to judge whether the data synchronization information meets the condition of issuing the synchronization information; if the publishing fails, judging whether the retry frequency of the publishing reaches the maximum value, and if the retry frequency of the publishing does not exceed the maximum value, reusing the database to publish the data synchronization information; if the retry number reaches the maximum value, an alarm is sent out to let the user know that the issue fails, and the user returns to the default idle state to wait for the next period.
As shown in FIG. 2, a process flow is described for encountering exceptions during the publication process; in the starting state, judging whether data in the last rows in the data synchronization information table are not released, if so, indicating that the data are abnormal in the releasing process, entering a releasing flow; if not, the method returns to the starting state again, and the control and processing of the exception embody the reliability of the method.
As shown in fig. 3, a process of acquiring data synchronization information and putting the acquired data synchronization information into a level 1 cache; the software is in an idle state by default, periodically judges whether other servers have data and files which need to be synchronized by the software, returns to the idle state if the other servers do not have the data and the files, and waits for the next period for judgment; if yes, judging whether the own cache region has no data and the space of the cache region is enough, and if the condition is met, acquiring the data needing to be synchronized on other servers into a level 1 cache region of the local computer; if the condition is not met, the method returns to the default idle state and waits for the next period.
As shown in fig. 4, a flow of acquiring data synchronization information, and putting the acquired data synchronization information into a level 2 cache for synchronization; the software is in an idle state by default, whether an idle space exists in the level 2 cache region is periodically judged, and if not, the software returns to the default idle state to wait for the next period for judgment; if yes, judging whether the level 1 cache region has data, if not, returning to a default idle state, if yes, acquiring the level 2 cache region from the level 1 cache region, synchronizing the information, updating a data synchronization release table after the synchronization, acquiring a file, and deleting the information from the level 1 cache region and the level 2 cache region, wherein the synchronization file can adopt FTP or other modes; after the information of the level 1 cache region and the level 2 cache region is deleted, the system returns to the default idle state, when the data volume needing to be synchronized is too large, if the synchronization speed is uncontrollable, a large amount of server resources are occupied, and the normal operation of other applications on the server is influenced; for this reason, the method uses a double-cache mechanism, and the synchronization speed is controllable, when the data amount required to be synchronized exceeds a certain threshold (the threshold is configured according to the server processing capacity), the period of data synchronization is adjusted, the data synchronization is acquired, a two-level cache is set in the cache region, the data is acquired from the synchronization source and then is put into the first-level cache, the data is sequentially taken out from the first-level cache region and then is put into the second-level cache, and after the synchronization process is completed, the data is deleted from the two-level cache.
As shown in fig. 5, the flow of deleting data synchronization information; the software is in an idle state by default, whether the data in the local information release table is synchronized is periodically judged, if so, the data is deleted and the software returns to the idle state; if not, judging whether the storage time of the data exceeds the storage period, and if not, returning to the idle state; if the storage time limit is exceeded, an alarm is sent out, the piece of data is deleted, and the state returns to the idle state.
The data synchronization process comprises the steps of releasing, obtaining and deleting, the method considers various possible exceptions in the three processes, when the synchronous data are written in and deleted from the database, failures can occur, and each type of database has a processing method for processing the exceptions; such as relational database SQL SERVER, which uses transactions to guarantee ACID properties; ACID properties, namely atomicity (which is a thing, done or not done, no intermediate interruption occurs, and once an error occurs, the state before the transaction starts is returned), consistency (data integrity is not damaged before and after the transaction starts), isolation (concurrent transactions are allowed to read and write data simultaneously, and the data inconsistency caused by the cross execution of a plurality of transactions is prevented), and persistence (once the transaction is completed, the modification of the data is permanent, and the data cannot be lost even if a system fails).
As shown in fig. 6, in order to support extension, support replacement of a database type within a small change range or without change, an abstract factory pattern in a design pattern is used to access a database, so that flexibility of access is increased, and extension is easier.
As shown in fig. 7, in one embodiment of the present invention, the data synchronization and file synchronization method is applied to a recording system, in which a main server and a standby server are synchronized, and a database and a file between the servers are backed up each other, thereby indicating high availability; the recording software main and recording software standby and data file synchronization all access the database, insert, inquire and delete the synchronous data, and realize the file synchronization through the FTP service.
As shown in fig. 8, the software of the present invention is applied to the internal composition block diagram of the recording system, acquires the synchronization information from the database module, puts the synchronization information into the buffer module, notifies the file management module to perform file synchronization, and notifies the database module to perform data synchronization; in addition, the three modules interact with the LOG module; it can be seen from the figure that one of the features of the present invention, file synchronization and database synchronization are associated, and synchronization of data and files is realized. The method has high reusability, and can be used in various use scenes, such as data synchronization and file synchronization of sound recording and video recording information, log data synchronization of network management and the like.
Example 1;
taking the case that the synchronization method is applied to a sound recording and video recording system, a plurality of servers are used in the sound recording and video recording system to record and record the call and the video in the system, only one main server is used for recording and video recording at the same time, and other standby servers need to synchronize the recorded information (stored in a database) of the recorded audio and video and the files of the audio and video; the identity of the main server can be switched to any standby server at any time, and a new main server is used for recording the call and the video in the system.
The method is specifically applied to actual projects, and a recording server is taken as an example. A plurality of (> = 2) sound recording servers operate simultaneously, call recording and sound recording audio on each sound recording server are required to be synchronous, a sound recording and video recording program and a data synchronization program are operated on each server, and the synchronization process of each sound recording and video recording data comprises the following processes: publishing, acquiring and deleting.
(1) Releasing synchronous information;
the main server recording program adds new data information into a data synchronous release table of a local database, and the data content comprises a recording initiator, a recording receiver, a recording number, a recording duration and a recording storage path;
issuing, namely adding contents into the form TRecordSync, wherein 2 schemes are added by which program for consideration, one scheme is that a main recording server program completes adding TRecordSync form contents, and the other scheme is that the main recording server program informs a data synchronization program, and a local data synchronization program adds TRecordSync form contents; in fact, the more intermediate steps, the more exceptions that may need to be considered, and in any case, in order to solve the problem that the record addition by the TRecordSync fails, a column of "whether the record has been issued" needs to be added to the trecordd table, when the recording server adds a call record to the trecordd, the column is assigned with a value of 0 to indicate that the record has not been issued, and when the record addition by the TRecordSync succeeds, the column is assigned with a value of 1 to indicate that the record has been issued.
(2) Acquiring synchronous information;
the data synchronization program periodically queries data synchronization release tables of other server databases to see whether data information which is not acquired by the local server exists, if so, the data synchronization release tables are inserted into a local data record table to copy audio and video, and the server which releases the call information knows that the standby server has been acquired;
when the data synchronization program acquires the call related information needing synchronization from other servers, the following steps are divided:
1) Querying another server database table TRecordSync, and if call information needing local synchronization exists, acquiring the call information;
2) Adding the call information into a local database table TRecord;
3) Audio and video indicated by the call information are copied to the local;
4) Setting the column of the table TRecordSync in which the local machine completes synchronization as 1 to indicate that the synchronization is performed;
after the synchronization information is obtained, the data corresponding to the synchronization information needs to be sent to a file synchronization module
And the piece synchronization module acquires the audio and video files from the synchronization source according to the information, and the synchronization can be realized through an FTP protocol.
(3) Deleting the synchronous information;
after the data information needing to be synchronized is issued, periodically inquiring whether the call information of the data synchronization issuing table of the local database is acquired by other servers, and if the call information is acquired, deleting the record; because the deletion of the TRecordSync form content is based on the premise that the call information is already synchronized by all servers, if the servers are not powered on for a long time, the record is not synchronized by all servers and is always left in the TRecordSync form, a time limit of 90 days at most needs to be set for the TRecordSync form content, and the record in the TRecordSync form is deleted if the record is not synchronized after the time;
through the three steps of releasing, obtaining and deleting, the application of the method in the recording system is realized, and data and file synchronization is provided for other server software and applications by taking the database as a synchronization medium.
Example 2
Taking the application of the synchronization method in a network management system as an example, the network management system records the alarm information in the system by using a plurality of servers, only one main server records the alarm information at the same time, and other standby servers need to synchronize the alarm information; the identity of the main server may be switched to any one of the standby servers at any time, and the new main server records the alarm information in the system.
The method is specifically applied to actual projects, and a network management server is taken as an example. And a plurality of (> = 2) network management servers operate simultaneously, and the alarm records on each network management server are required to be synchronous. Each server runs a network management program and a data synchronization program, and the synchronization process of each alarm data comprises the following processes: publishing, acquiring and deleting.
(1) Releasing synchronous information;
the main server network management program adds new data information into a data synchronous release table of a local database, and the data content comprises an alarm type, an alarm level, an alarm source and alarm content;
issuing, namely adding contents into the table TalamSy, and considering 2 schemes according to which program is added, wherein one scheme is that the main network management server program completes adding the table contents of the TalamSy, and the other scheme is that the main network management server program informs a data synchronization program, and a local data synchronization program adds the table contents of the TalamSy; in fact, the more intermediate steps, the more exceptions that may need to be considered, and in any scheme, in order to solve the problem that the addition of the record by the talamsync fails, a column of "whether the record has been issued" needs to be added in the TAlarm table, and when the webmaster server adds the alarm record to TAlarm, the value of the column is assigned to 0 to indicate that the record has not been issued, and when the addition of the record by the talamsync succeeds, the value of the column is assigned to 1 to indicate that the record has been issued.
(2) Acquiring synchronous information;
the data synchronization program periodically queries data synchronization release tables of other server databases to see whether data information which is not acquired by the local server exists, if so, the data synchronization release tables are inserted into a local data record table, and the server which releases the alarm information knows that the standby server has been acquired;
when the data synchronization program acquires the alarm information needing synchronization from other servers, the following steps are carried out:
1) Querying another server database table TAlarmSync, and if alarm information needing local synchronization exists, acquiring the alarm information;
2) Adding the alarm information into a local database table TAlarm;
3) Setting the column of the table TAlarmSync for completing the local synchronization as 1 to show that the synchronization is finished;
(3) Deleting synchronization information
After the data information needing to be synchronized is issued, periodically inquiring whether the alarm information of the data synchronization issuing table of the local database is acquired by other servers, and if the alarm information is acquired, deleting the record; because the premise of deleting the talamsync table content is that the alarm information is already completed by all servers in synchronization, if the server is not powered on for a long time, the record is not completed by all servers in synchronization, and the record is always left in the talamsync table, a time limit of 90 days at most needs to be set for the talamsync table content, and the record in the talamsync table is deleted if the record is not synchronized after the time;
the method is applied to a network management system through the three steps of releasing, obtaining and deleting, and data synchronization is provided for other server software and applications by taking the database as a synchronization medium.
Claims (2)
1. A method for synchronizing high-multiplexing extensible data and files is characterized in that: the method takes a server or PC equipment as a carrier and an operating system as windows or linux to realize the synchronization of data and files among a plurality of pieces of equipment, and comprises the following steps:
firstly, releasing synchronous information;
data information needing to be synchronized, including time and data content, is published, each server is used as a synchronization source to publish the data, and the publishing method is that the data information is written into a database table, and the type of the database supports a relational database, a non-relational database and a key value database;
secondly, acquiring synchronous information;
acquiring synchronous information from other servers, periodically inquiring whether synchronous data are issued on the other servers, if so, acquiring and updating the synchronous information to a local database, and if not, waiting for the next period to inquire;
thirdly, deleting the synchronous information;
after the data information needing to be synchronized is issued, periodically inquiring whether the data content of a data synchronization issuing table of the local database is acquired by other servers, if so, deleting the data synchronization information record, and if not, waiting for the next period for inquiry;
through the three steps, each server can serve as a synchronization source to release information and simultaneously acquire information from other servers, the synchronization information released by each server is deleted after the synchronization information is acquired, and the database serves as a synchronization medium to provide synchronization of data and files for other servers;
the data content in the first step is that when the data content is applied to different systems, the data content is self-defined according to system requirements, and when the data content is applied to a recording system, the data content comprises a recording initiator, a recording receiver, a recording number, a recording duration and a recording storage path; when the method is applied to a network management system, the data content comprises an alarm type, an alarm level, an alarm source and alarm content;
the method for writing data information into a database table described in the first step is as follows:
the sentence patterns used are different according to different types of the database;
the non-relational database MySQL uses the sentence form as follows:
INSERT INTOs table name (field 1, field 2.. Field n) VALUES (value 1, value 2.. Value n);
the relational database SQLServer uses a sentence pattern of:
update table name set column name = value;
the key value database MongoDB uses sentence patterns as follows:
db. sets the name insertOne ({ key name: value });
the method for periodically inquiring whether synchronous data are published on other servers in the second step is as follows:
the query method uses different sentence patterns according to different types of the database:
the non-relational database MySQL uses the sentence form as follows:
selecting distinting from 'table name' where 'restriction condition' group by 'group, displaying the number of strips' according to 'having' filtering condition 'order by limit';
the relational database SQLServer uses a sentence pattern of:
select from table name where column name = 'value';
the key value database MongoDB uses sentence patterns as follows:
db. sets the name find ({ conditional document });
after the data is inquired, the data is inserted into a local server database, the inquired data is firstly put into a local cache in the inquiry insertion process, then the data is taken out from the local cache and inserted into the local database, in order to more efficiently and more flexibly adjust the synchronization speed according to the data volume, a double-level cache mechanism is introduced and the synchronization speed is changed according to the data volume needing to be synchronized, so that the flexible synchronization process is realized, after the data is acquired, a file synchronization module is informed, the file synchronization is performed according to the value of the data, and the file is acquired from a synchronization source by using an FTP protocol;
the method for deleting the data synchronization information record in the third step is as follows:
due to different database types, the operation statements are different:
a non-relational database MySQL, which uses a sentence pattern of MySQL;
DELETE FROM < table name > [ WHERE clause ] [ ORDER BY clause ] [ LIMIT clause ]
The relational database SQLServer uses the sentence pattern as follows:
delete from table name where column name = 'value';
the key value database MongoDB uses sentence patterns as follows:
db. sets the name remove ({ column name: value });
the synchronization information is deleted from the database table using a deletion schema.
2. The method of claim 1, wherein the method further comprises:
the method for introducing the double-stage cache mechanism is that the synchronization speed is changed according to the size of the data volume to be synchronized, and the flexible synchronization process is realized by the following steps:
after data synchronization information is obtained, the data synchronization information is placed into a level 1 cache, then the data is placed into a level 2 cache from the level 1 cache, the data synchronization information is obtained from a level 2 cache region and is written into a database, after the data is successfully written, the data synchronization information is deleted from the level 1 cache region and the level 2 cache region, a double-level cache mechanism is adopted, when the data quantity needing to be synchronized is overlarge, the synchronization speed is controllable, and when the data quantity needing to be synchronized is inquired to exceed a threshold value, the period of inquiring and obtaining the data synchronization is adjusted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210676804.8A CN114756628B (en) | 2022-06-16 | 2022-06-16 | High-multiplexing extensible data and file synchronization method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210676804.8A CN114756628B (en) | 2022-06-16 | 2022-06-16 | High-multiplexing extensible data and file synchronization method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114756628A CN114756628A (en) | 2022-07-15 |
| CN114756628B true CN114756628B (en) | 2022-10-28 |
Family
ID=82337046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210676804.8A Active CN114756628B (en) | 2022-06-16 | 2022-06-16 | High-multiplexing extensible data and file synchronization method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114756628B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103823797A (en) * | 2012-11-16 | 2014-05-28 | 镇江诺尼基智能技术有限公司 | FTP (file transfer protocol) based real-time industry database data synchronization system |
| CN103839191A (en) * | 2014-02-21 | 2014-06-04 | 国家电网公司 | Method for integrally and synchronously maintaining custom-made model |
| CN103970833A (en) * | 2014-04-02 | 2014-08-06 | 浙江大学 | Method for achieving two-way synchronous data circulation in heterogeneous database synchronizing system based on logs |
| CN106919574A (en) * | 2015-12-24 | 2017-07-04 | 中国科学院深圳先进技术研究院 | A kind of method of real-time processing remote synchronization file |
| CN107301196A (en) * | 2017-05-03 | 2017-10-27 | 北京小度信息科技有限公司 | Method of data synchronization, device, equipment and monitoring server |
| CN108021674A (en) * | 2017-12-06 | 2018-05-11 | 浙江远算云计算有限公司 | A kind of multi-level buffer delivery acceleration system of synchronization high in the clouds emulation data |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN112035463A (en) * | 2020-07-22 | 2020-12-04 | 武汉达梦数据库有限公司 | Bidirectional synchronization method and synchronization device of heterogeneous database based on log analysis |
| CN112988897A (en) * | 2021-03-30 | 2021-06-18 | 工银科技有限公司 | Data bidirectional synchronization method and device in system upgrading scene |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11200255B2 (en) * | 2019-08-22 | 2021-12-14 | Adp, Llc | Robust data synchronization solution between databases |
-
2022
- 2022-06-16 CN CN202210676804.8A patent/CN114756628B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103823797A (en) * | 2012-11-16 | 2014-05-28 | 镇江诺尼基智能技术有限公司 | FTP (file transfer protocol) based real-time industry database data synchronization system |
| CN103839191A (en) * | 2014-02-21 | 2014-06-04 | 国家电网公司 | Method for integrally and synchronously maintaining custom-made model |
| CN103970833A (en) * | 2014-04-02 | 2014-08-06 | 浙江大学 | Method for achieving two-way synchronous data circulation in heterogeneous database synchronizing system based on logs |
| CN106919574A (en) * | 2015-12-24 | 2017-07-04 | 中国科学院深圳先进技术研究院 | A kind of method of real-time processing remote synchronization file |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN107301196A (en) * | 2017-05-03 | 2017-10-27 | 北京小度信息科技有限公司 | Method of data synchronization, device, equipment and monitoring server |
| CN108021674A (en) * | 2017-12-06 | 2018-05-11 | 浙江远算云计算有限公司 | A kind of multi-level buffer delivery acceleration system of synchronization high in the clouds emulation data |
| CN112035463A (en) * | 2020-07-22 | 2020-12-04 | 武汉达梦数据库有限公司 | Bidirectional synchronization method and synchronization device of heterogeneous database based on log analysis |
| CN112988897A (en) * | 2021-03-30 | 2021-06-18 | 工银科技有限公司 | Data bidirectional synchronization method and device in system upgrading scene |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114756628A (en) | 2022-07-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110249321B (en) | System and method for capturing change data from a distributed data source for use by heterogeneous targets | |
| US10078682B2 (en) | Differentiated secondary index maintenance in log structured NoSQL data stores | |
| CN105956176B (en) | Database management system | |
| US6873995B2 (en) | Method, system, and program product for transaction management in a distributed content management application | |
| US6615223B1 (en) | Method and system for data replication | |
| US8756196B2 (en) | Propagating tables while preserving cyclic foreign key relationships | |
| US8214377B2 (en) | Method, system, and program for managing groups of objects when there are different group types | |
| US20070174325A1 (en) | Method and system for building a database from backup data images | |
| US7472125B2 (en) | Method for managing a database system | |
| CN108509462B (en) | Method and device for synchronizing activity transaction table | |
| CN106570163B (en) | Audit log read-write management method and system facing unreliable environment | |
| US20190347167A1 (en) | Primary Node-Standby Node Data Transmission Method, Control Node, and Database System | |
| US8214376B1 (en) | Techniques for global single instance segment-based indexing for backup data | |
| KR20060004915A (en) | Recovery from failures within data processing systems | |
| CN111414424B (en) | Method, system, medium and equipment for automatically synchronizing redis of configuration data | |
| KR20190063835A (en) | System for processing real-time data modification of in-memory database | |
| US20180276267A1 (en) | Methods and system for efficiently performing eventual and transactional edits on distributed metadata in an object storage system | |
| CN114756628B (en) | High-multiplexing extensible data and file synchronization method | |
| CN114564458B (en) | Method, device, equipment and storage medium for synchronizing data among clusters | |
| CN116821232A (en) | Data synchronization method and related device | |
| CN114116717A (en) | Database information query method, device, equipment and readable storage medium | |
| CN113076298A (en) | Distributed small file storage system | |
| CN111522688A (en) | Data backup method and device for distributed system | |
| CN113672683B (en) | Spark SQL-based distributed database metadata synchronization device and method | |
| CN117131023B (en) | Data table processing method, device, computer equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |