CN114756628A - High-multiplexing extensible data and file synchronization method - Google Patents
High-multiplexing extensible data and file synchronization method Download PDFInfo
- Publication number
- CN114756628A CN114756628A CN202210676804.8A CN202210676804A CN114756628A CN 114756628 A CN114756628 A CN 114756628A CN 202210676804 A CN202210676804 A CN 202210676804A CN 114756628 A CN114756628 A CN 114756628A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronization
- information
- database
- servers
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 230000001360 synchronised effect Effects 0.000 claims abstract description 65
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 239000003999 initiator Substances 0.000 claims description 3
- 238000003780 insertion Methods 0.000 claims description 3
- 230000037431 insertion Effects 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 239000002253 acid Substances 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A method for synchronizing high-multiplexing extensible data and files takes a server or a PC as a carrier and comprises the following steps: releasing synchronous information, namely releasing data information needing to be synchronized, wherein each server is used as a synchronous source to release data; acquiring synchronous information, and acquiring the synchronous information from other servers; deleting synchronous information, periodically inquiring whether the call information of the data synchronous release table of the local database is acquired by other servers, if so, deleting data synchronous information records, and if not, waiting for the next period for inquiry; through the three steps, each server issues information, simultaneously acquires information from other servers, and provides data and file synchronization for other software and applications by taking the database as a synchronization medium. The invention realizes mutual synchronization among a plurality of servers, and realizes file synchronization and database synchronization among a plurality of servers.
Description
Technical Field
The invention relates to the technical field of database and file synchronization of multiple servers, in particular to a high-multiplexing extensible data and file synchronization method.
Background
The traditional data synchronization method in the computer system mainly adopts a synchronization mechanism of database software. For example, SQL Server, which uses a publish-subscribe method to achieve database synchronization, publishes messages by a main Server, subscribes by a backup Server, publishes messages when data of the main Server changes, the backup Server reads messages to perform synchronous update, and the delay of the intermediate process is short.
Problems with this approach are:
(1) when the data quantity to be synchronized is large, the data synchronization method can cause the blocking of the server and influence the normal operation of various programs and services on the server.
(2) A large number of updates or insertions to the publish database may occur with errors or inconsistent data from the subscription servers.
(3) When the number of devices needing synchronization increases, the number of newly-added devices and the number of devices in the original synchronization system need to increase the number of issuing and subscribing transactions, and the deployment is tedious.
(4) Scalability is poor, especially when data synchronization is associated with other services, such as service requirements that require data synchronization to be associated with file synchronization, and the publish-subscribe method of the database cannot be associated with file synchronization.
The above is a traditional database synchronization method, while the traditional file synchronization method generally adopts tools such as rsync or synchronous disk, and if the data volume is small or the requirement on timeliness is not so high, the requirement can be basically met. However, when the amount of data is large, the speed, reliability, stability and timeliness of these conventional synchronization tools are not always sufficient to meet the business requirements.
Taking patent "a data synchronization method, system and storage medium" as an example, publication No. CN110377602A is a traditional data synchronization method, which also takes a database as a medium, and triggers generation of trigger information and sends the trigger information to a corresponding database table when monitoring a synchronous data sending event operation of a master database. The data synchronization method is suitable for a simple data synchronization scene, a main server is used as a data source, the expansibility and the high availability are poor, and when the main server breaks down, the synchronization process is stagnated.
In the patent "a distributed database data synchronization method", publication No. CN113918656B, a central server is used as a synchronization center to receive and issue synchronization information, and this synchronization method requires high requirements on stability and high availability of the central server, and has no explicit processing method for large data synchronization requirements, which may cause server blocking in abnormal situations.
Disclosure of Invention
In view of the problems of unsmooth synchronization, complicated synchronization steps of a plurality of servers and poor expansibility in the prior art, the invention provides a method for synchronizing high-multiplexing extensible data and files.
The technical scheme adopted by the invention is as follows: a method for synchronizing high-multiplexing extensible data and files takes a server or PC equipment as a carrier and an operating system as windows or linux, realizes the synchronization of the data and the files among a plurality of equipment, and comprises the following specific steps:
step one, synchronous information is issued;
data information needing to be synchronized, including time and data content, is published, each server is used as a synchronization source to publish the data, and the publishing method is that the data information is written into a database table, and the type of the database supports a relational database, a non-relational database and a key value database;
secondly, acquiring synchronous information;
acquiring synchronous information from other servers, periodically inquiring whether synchronous data are issued on the other servers, if so, acquiring and updating the synchronous information into a local database, and if not, waiting for the next period to inquire;
thirdly, deleting the synchronous information;
after the data information needing to be synchronized is issued, periodically inquiring whether the data content of a data synchronization issuing table of the local database is acquired by other servers, if so, deleting the data synchronization information record, and if not, waiting for the next period for inquiry;
through the three steps, each server can serve as a synchronization source to issue information and simultaneously acquire information from other servers, the synchronization information issued by each server is deleted after the synchronization information is acquired, and the database serves as a synchronization medium to provide synchronization of data and files for other servers.
The data content in the first step is that when the data content is applied to different systems, the data content is self-defined according to system requirements, and when the data content is applied to a recording system, the data content comprises a recording initiator, a recording receiver, a recording number, a recording duration and a recording storage path; when the method is applied to a network management system, the data content comprises an alarm type, an alarm level, an alarm source and alarm content.
The method for writing data information into a database table described in the first step is as follows:
the sentence patterns used are different according to different types of the database;
the non-relational database MySQL uses the sentence expression as follows:
INSERT INTOs table name (field 1, field 2.. field n) VALUES (value 1, value 2.. value n);
the relational database SQLServer uses the sentence pattern as follows:
update table name set column name = value;
the key value database MongoDB uses sentence patterns as follows:
db. collection name insertOne ({ key name: value }).
The method for periodically inquiring whether synchronous data are published on other servers in the second step is as follows
The query method uses different sentence patterns according to different types of databases:
the non-relational database MySQL uses the sentence expression as follows:
selecting distinting from 'table name' where 'constraint' group by 'group shows number' according to 'having' filtering condition 'order bylimit';
the relational database SQLServer uses a sentence pattern of:
select from table name where column name = 'value';
the key value database MongoDB uses sentence patterns as follows:
db. set name find ({ condition document });
after the data is inquired, the data is inserted into a local server database, the inquired data is firstly put into a local cache in the inquiry insertion process, then the data is taken out from the local cache and inserted into the local database, in order to more efficiently and more flexibly adjust the synchronization speed according to the data volume, a double-level cache mechanism is introduced and the synchronization speed is changed according to the data volume needing to be synchronized, so that the flexible synchronization process is realized, after the data is acquired, a file synchronization module is informed, the file synchronization is performed according to the value of the data, and the file is acquired from a synchronization source by using an FTP protocol.
The method for deleting the data synchronization information record in the third step is as follows:
due to different database types, the operation statements are different:
a non-relational database MySQL, which uses a sentence pattern of MySQL;
DELETE FROM < table name > [ WHERE clause ] [ ORDER BY clause ] [ LIMIT clause ]
The relational database SQLServer uses a sentence pattern of:
delete from table name where column name = 'value';
the key value database MongoDB uses sentence patterns as follows:
db. set name remove ({ column name: value });
the synchronization information is deleted from the database table using a deletion schema.
The method for introducing the double-stage cache mechanism is that the synchronization speed is changed according to the size of the data volume to be synchronized, and the flexible synchronization process is realized by the following steps:
after data synchronization information is obtained, the data synchronization information is placed into a level 1 cache, then the data is placed into a level 2 cache from the level 1 cache, the data synchronization information is obtained from a level 2 cache region and is written into a database, after the data is successfully written, the data synchronization information is deleted from the level 1 cache region and the level 2 cache region, a double-level cache mechanism is adopted, when the data quantity needing to be synchronized is overlarge, the synchronization speed is controllable, and when the data quantity needing to be synchronized is inquired to exceed a threshold value, the period of inquiring and obtaining the data synchronization is adjusted.
The invention has the beneficial effects that: the method and the system realize the correlation of data and file synchronization, solve the problems of unsmooth synchronization, complicated synchronization steps of a plurality of servers and poor expansibility in the traditional data synchronization and file synchronization, have controllable and stable synchronization speed, are easy to expand, are suitable for a plurality of scenes needing data and file synchronization, and have the characteristics of high reusability and expandability.
The multiple servers (> = 2) are mutually synchronous sources, mutual synchronization among the multiple servers can be realized, file synchronization and database synchronization among the multiple servers can be realized, and the method can be applied to multiple operating systems such as windows and linux.
Drawings
FIG. 1 is a flow chart of data synchronization information distribution according to the present invention;
FIG. 2 is a process flow of the exception handling of the data synchronization information issue according to the present invention;
FIG. 3 is a flow of data synchronization information acquisition into level 1 cache according to the present invention;
FIG. 4 is a flow of data synchronization information acquisition into level 2 cache according to the present invention;
FIG. 5 is a flow chart of deleting data synchronization information according to the present invention;
FIG. 6 is a block diagram of the interface class diagram design of the present invention extensible to support multiple databases;
FIG. 7 is a block diagram of an external connection for synchronizing data files in a recording system according to the present invention;
FIG. 8 is a block diagram of the internal components of the recording system for synchronizing data files according to the present invention.
Detailed Description
As shown in fig. 1, the process of issuing data synchronization information: the software is in an idle state by default, periodically judges whether the software accords with the condition of releasing the synchronous information, and if the software accords with the condition, the software uses a database to release the data synchronous information; if not, continuing to periodically judge whether the self meets the condition of releasing the synchronous information; after the data synchronization information is issued, judging whether the data synchronization information is successfully issued, if the data synchronization information is successfully issued, returning to a default idle state, and waiting for the next period to judge whether the data synchronization information meets the condition of issuing the synchronization information; if the release fails, judging whether the retry frequency of the release reaches the maximum value, if not, reusing the database to release the data synchronization information; if the retry number reaches the maximum value, an alarm is sent out to let the user know that the issue fails, and the user returns to the default idle state to wait for the next period.
As shown in FIG. 2, a process flow is described for encountering exceptions during the publication process; in the starting state, judging whether data in the last rows in the data synchronization information table are not released, if so, indicating that the data are abnormal in the releasing process, entering a releasing flow; if not, the method returns to the starting state again, and the control and processing of the exception embody the reliability of the method.
As shown in fig. 3, a process of acquiring data synchronization information and placing the acquired data synchronization information into a level 1 cache; the software is in an idle state by default, periodically judges whether other servers have data and files which need to be synchronized by the software, returns to the idle state if the other servers do not have the data and the files, and waits for the next period for judgment; if yes, judging whether the own cache region has no data and the space of the cache region is enough, and if the condition is met, acquiring the data needing to be synchronized on other servers into a level 1 cache region of the local computer; and if the condition is not met, returning to a default idle state and waiting for the next period.
As shown in fig. 4, a flow of acquiring data synchronization information, and putting the acquired data synchronization information into a level 2 cache for synchronization; the software is in an idle state by default, whether an idle space exists in the level 2 cache region is periodically judged, and if not, the software returns to the default idle state to wait for the next period for judgment; if yes, judging whether the level 1 cache region has data, if not, returning to a default idle state, if yes, acquiring the level 2 cache region from the level 1 cache region, synchronizing the information, updating a data synchronization release table after the synchronization, acquiring a file, and deleting the information from the level 1 cache region and the level 2 cache region, wherein the synchronization file can adopt FTP or other modes; after the information of the level 1 cache region and the level 2 cache region is deleted, the system returns to a default idle state, when the data volume needing to be synchronized is too large, if the synchronization speed is uncontrollable, a large amount of server resources are occupied, and the normal operation of other applications on the server is influenced; for this reason, the method uses a double-cache mechanism, and the synchronization speed is controllable, when the data amount required to be synchronized exceeds a certain threshold (the threshold is configured according to the server processing capacity), the period of data synchronization is adjusted, the data synchronization is acquired, a two-level cache is set in the cache region, the data is acquired from the synchronization source and then is put into the first-level cache, the data is sequentially taken out from the first-level cache region and then is put into the second-level cache, and after the synchronization process is completed, the data is deleted from the two-level cache.
As shown in fig. 5, the flow of deleting data synchronization information; the software is in an idle state by default, whether the data in the local information release table is synchronized is periodically judged, if so, the data is deleted and the software returns to the idle state; if not, judging whether the storage time of the data exceeds the storage period, and if not, returning to the idle state; if the storage time limit is exceeded, an alarm is sent out, the piece of data is deleted, and the state returns to the idle state.
The data synchronization process comprises the steps of releasing, obtaining and deleting, the method considers various possible exceptions in the three processes, when the synchronous data are written in and deleted from the database, failures can occur, and each type of database has a processing method for processing the exceptions; such as a relational database SQL SERVER that uses transactions to guarantee ACID characteristics; ACID properties, namely atomicity (which is a thing, done or not done, no intermediate interruption occurs, and once an error occurs, the state before the transaction starts is returned), consistency (data integrity is not damaged before and after the transaction starts), isolation (concurrent transactions are allowed to read and write data simultaneously, and multiple transactions are prevented from being executed in a cross manner to cause data inconsistency), and persistence (once a transaction is completed, modification of data is permanent, and the data cannot be lost even if a system fails).
As shown in fig. 6, in order to support extension, support replacement of a database type within a small change range or without change, an abstract factory pattern in a design pattern is used to access a database, so that flexibility of access is increased, and extension is easier.
As shown in fig. 7, in one embodiment of the present invention, the data synchronization and file synchronization method is applied to a recording system, in which a main server and a standby server are synchronized, and a database and a file between the servers are backed up each other, thereby indicating high availability; the recording software main and recording software standby and data file synchronization all access the database, insert, inquire and delete the synchronous data, and realize the file synchronization through the FTP service.
As shown in fig. 8, the software of the present invention is applied to the internal composition block diagram of the recording system, acquires the synchronization information from the database module, puts the synchronization information into the buffer module, notifies the file management module to perform file synchronization, and notifies the database module to perform data synchronization; in addition, the three modules interact with the LOG module; one of the features of the present invention, file synchronization and database synchronization are associated to achieve synchronization of data and files. The method has high reusability, and can be used in various use scenes, such as data synchronization and file synchronization of sound recording and video recording information, log data synchronization of network management and the like.
Example 1;
taking the example that the synchronization method is applied to a recording and video recording system, a plurality of servers are used in the recording and video recording system to record and record the call and the video in the system, only one main server records and video at the same time, and other standby servers need to synchronize the recorded recording information (stored in a database) of the recorded audio and video and the files of the audio and video; the identity of the main server can be switched to any standby server at any time, and a new main server is used for recording the call and the video in the system.
The method is specifically applied to actual projects, and a recording server is taken as an example. A plurality of (> = 2) sound recording servers operate simultaneously, call recording and sound recording audio on each sound recording server are required to be synchronous, a sound recording and video recording program and a data synchronization program are operated on each server, and the synchronization process of each sound recording and video recording data comprises the following processes: publishing, acquiring and deleting.
(1) Releasing synchronous information;
the main server audio recording program adds new data information into a data synchronous release table of a local database, and the data content comprises an audio recording initiator, an audio recording receiver, an audio recording number, an audio recording duration and an audio recording storage path;
issuing, namely adding contents into the form TRecordSync, wherein 2 schemes are added by which program for consideration, one scheme is that a main recording server program completes adding TRecordSync form contents, and the other scheme is that the main recording server program informs a data synchronization program, and a local data synchronization program adds TRecordSync form contents; in fact, the more intermediate steps, the more exceptions that may need to be considered, and in any case, in order to solve the problem that the record addition by the TRecordSync fails, a column of "whether the record has been issued" needs to be added to the trecordd table, when the recording server adds a call record to the trecordd, the column is assigned with a value of 0 to indicate that the record has not been issued, and when the record addition by the TRecordSync succeeds, the column is assigned with a value of 1 to indicate that the record has been issued.
(2) Acquiring synchronous information;
the data synchronization program periodically queries data synchronization release tables of other server databases to see whether data information which is not acquired by the local server exists, if so, the data synchronization release tables are inserted into a local data record table to copy audio and video, and the server which releases the call information knows that the standby server has been acquired;
when the data synchronization program acquires the call related information needing synchronization from other servers, the data synchronization program comprises the following steps:
1) querying another server database table TRecordSync, and if call information needing local synchronization exists, acquiring the call information;
2) adding the call information into a local database table TRecord;
3) copying the audio and video indicated by the call information to the local;
4) setting the column of the table TRecordSync in which the local machine completes synchronization as 1 to indicate that the synchronization is performed;
after the synchronization information is obtained, the data corresponding to the synchronization information needs to be sent to a file synchronization module
And the piece synchronization module acquires the audio and video files from the synchronization source according to the information, and the synchronization can be realized through an FTP protocol.
(3) Deleting the synchronous information;
after the data information needing to be synchronized is issued, periodically inquiring whether the call information of the data synchronization issuing table of the local database is acquired by other servers, and if the call information is acquired, deleting the record; because the deletion of the TRecordSync form content is based on the premise that the call information is already synchronized by all servers, if the servers are not powered on for a long time, the record is not synchronized by all servers and is always left in the TRecordSync form, a time limit of 90 days at most needs to be set for the TRecordSync form content, and the record in the TRecordSync form is deleted if the record is not synchronized after the time;
through the three steps of releasing, obtaining and deleting, the application of the method in the recording system is realized, and data and file synchronization is provided for other server software and applications by taking the database as a synchronization medium.
Example 2
Taking the application of the synchronization method in a network management system as an example, the network management system records the alarm information in the system by using a plurality of servers, only one main server records the alarm information at the same time, and other standby servers need to synchronize the alarm information; the identity of the main server may be switched to any one of the standby servers at any time, and the new main server records the alarm information in the system.
The method is specifically applied to actual projects, and a network management server is taken as an example. And a plurality of (> = 2) network management servers operate simultaneously, and the alarm records on each network management server are required to be synchronous. Each server runs a network management program and a data synchronization program, and the synchronization process of each alarm data comprises the following processes: publishing, acquiring and deleting.
(1) Releasing the synchronous information;
the master server network management program adds new data information into a data synchronous release table of a local database, and the data content comprises an alarm type, an alarm level, an alarm source and alarm content;
issuing, namely adding contents into the table TalamSy, and considering 2 schemes according to which program is added, wherein one scheme is that the main network management server program completes adding the table contents of the TalamSy, and the other scheme is that the main network management server program informs a data synchronization program, and a local data synchronization program adds the table contents of the TalamSy; in fact, the more intermediate steps, the more exceptions that may need to be considered, and in any scheme, in order to solve the problem that the addition of the record by the talamsync fails, a column of "whether the record has been issued" needs to be added in the TAlarm table, and when the webmaster server adds the alarm record to TAlarm, the value of the column is assigned to 0 to indicate that the record has not been issued, and when the addition of the record by the talamsync succeeds, the value of the column is assigned to 1 to indicate that the record has been issued.
(2) Acquiring synchronous information;
the data synchronization program periodically queries data synchronization release tables of other server databases to see whether data information which is not acquired by the local server exists, if so, the data synchronization release tables are inserted into a local data record table, and the server which releases the alarm information knows that the standby server has been acquired;
when the data synchronization program acquires the alarm information needing synchronization from other servers, the data synchronization program comprises the following steps:
1) querying another server database table TAlarmSync, and if alarm information needing local synchronization exists, acquiring the alarm information;
2) adding the alarm information into a local database table TAlarm;
3) setting the column of the table TAlarmSync for completing the local synchronization as 1 to show that the synchronization is finished;
(3) deleting synchronization information
After the data information needing to be synchronized is issued, periodically inquiring whether the alarm information of the data synchronization issuing table of the local database is acquired by other servers, and if the alarm information is acquired, deleting the record; because the premise of deleting the talamsync table content is that the alarm information is already completed by all servers in synchronization, if the server is not powered on for a long time, the record is not completed by all servers in synchronization, and the record is always left in the talamsync table, a time limit of 90 days at most needs to be set for the talamsync table content, and the record in the talamsync table is deleted if the record is not synchronized after the time;
through the three steps of releasing, obtaining and deleting, the application of the method in the network management system is realized, and data synchronization is provided for other server software and applications by taking the database as a synchronization medium.
Claims (6)
1. A method for synchronizing high-multiplexing extensible data and files is characterized in that: the method comprises the following steps of taking a server or PC equipment as a carrier and an operating system as windows or linux, realizing synchronization of data and files among a plurality of pieces of equipment, and specifically comprising the following steps:
step one, synchronous information is issued;
the data information needing to be synchronized, including time and data content, is published, each server is used as a synchronization source to publish the data, and the publishing method is to write the data information into a database table, wherein the database type supports a relational database, a non-relational database and a key value database;
secondly, acquiring synchronous information;
acquiring synchronous information from other servers, periodically inquiring whether synchronous data are issued on the other servers, if so, acquiring and updating the synchronous information to a local database, and if not, waiting for the next period to inquire;
thirdly, deleting the synchronous information;
after the data information needing to be synchronized is issued, periodically inquiring whether the data content of a data synchronization issuing table of the local database is acquired by other servers, if so, deleting the data synchronization information record, and if not, waiting for the next period for inquiry;
through the three steps, each server can serve as a synchronization source to issue information and simultaneously acquire information from other servers, the synchronization information issued by each server is deleted after the synchronization information is acquired, and the database serves as a synchronization medium to provide synchronization of data and files for other servers.
2. The method for highly multiplexed scalable data and file synchronization of claim 1, wherein: the data content in the first step is that when the data content is applied to different systems, the data content is self-defined according to system requirements, and when the data content is applied to a recording system, the data content comprises a recording initiator, a recording receiver, a recording number, a recording duration and a recording storage path; when the method is applied to a network management system, the data content comprises an alarm type, an alarm level, an alarm source and alarm content.
3. The method of claim 1, wherein the method further comprises:
the method for writing data information into a database table described in the first step is as follows:
the sentence patterns used are different according to different types of the database;
the non-relational database MySQL uses the sentence expression as follows:
INSERT INTOs table name (field 1, field 2.. field n) VALUES (value 1, value 2.. value n);
the relational database SQLServer uses a sentence pattern of:
update table name set column name = value;
the key value database MongoDB uses sentence patterns as follows:
db. collection name INSERTONE ({ Key name: value }).
4. The method of claim 1, wherein the method further comprises:
the method for periodically inquiring whether synchronous data are published on other servers in the second step is as follows
The query method uses different sentence patterns according to different types of databases:
the non-relational database MySQL uses the sentence expression as follows:
selecting distinting from 'table name' where 'constraint' group by 'group shows number' according to 'having' filtering condition 'order bylimit';
the relational database SQLServer uses a sentence pattern of:
select from table name where column name = 'value';
the key value database MongoDB uses sentence patterns as follows:
db. set name find ({ condition document });
after the data is inquired, the data is inserted into a local server database, the inquired data is firstly put into a local cache in the inquiry insertion process, then the data is taken out from the local cache and inserted into the local database, in order to more efficiently and more flexibly adjust the synchronization speed according to the data volume, a double-level cache mechanism is introduced and the synchronization speed is changed according to the data volume needing to be synchronized, so that the flexible synchronization process is realized, after the data is acquired, a file synchronization module is informed, the file synchronization is performed according to the value of the data, and the file is acquired from a synchronization source by using an FTP protocol.
5. The method for highly multiplexed scalable data and file synchronization of claim 1, wherein:
the method for deleting the data synchronization information record in the third step is as follows:
due to different database types, the operation statements are different:
a non-relational database MySQL, wherein the used sentence pattern is as follows;
DELETE FROM < table name > [ WHERE clause ] [ ORDER BY clause ] [ LIMIT clause ]
The relational database SQLServer uses a sentence pattern of:
delete from table name where column name = 'value';
the key value database MongoDB uses sentence patterns as follows:
db. set name remove ({ column name: value });
the synchronization information is deleted from the database table using a deletion schema.
6. The method of claim 4, wherein the method further comprises:
the method for introducing the double-stage cache mechanism is that the synchronization speed is changed according to the size of the data volume to be synchronized, and the flexible synchronization process is realized by the following steps:
after data synchronization information is obtained, the data synchronization information is put into a level 1 cache, then the data is put into a level 2 cache from the level 1 cache, the data synchronization information is obtained from a level 2 cache area and is written into a database, after the data synchronization information is successfully written, the data synchronization information is deleted from the level 1 cache area and the level 2 cache area, a double-level cache mechanism is adopted, when the data quantity needing to be synchronized is too large, the synchronization speed is controllable, and when the data quantity needing to be synchronized is inquired to exceed a threshold value, the period of inquiring and obtaining the data synchronization is adjusted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210676804.8A CN114756628B (en) | 2022-06-16 | 2022-06-16 | High-multiplexing extensible data and file synchronization method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210676804.8A CN114756628B (en) | 2022-06-16 | 2022-06-16 | High-multiplexing extensible data and file synchronization method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114756628A true CN114756628A (en) | 2022-07-15 |
| CN114756628B CN114756628B (en) | 2022-10-28 |
Family
ID=82337046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210676804.8A Active CN114756628B (en) | 2022-06-16 | 2022-06-16 | High-multiplexing extensible data and file synchronization method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114756628B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103823797A (en) * | 2012-11-16 | 2014-05-28 | 镇江诺尼基智能技术有限公司 | FTP (file transfer protocol) based real-time industry database data synchronization system |
| CN103839191A (en) * | 2014-02-21 | 2014-06-04 | 国家电网公司 | Method for integrally and synchronously maintaining custom-made model |
| CN103970833A (en) * | 2014-04-02 | 2014-08-06 | 浙江大学 | Method for achieving two-way synchronous data circulation in heterogeneous database synchronizing system based on logs |
| CN106919574A (en) * | 2015-12-24 | 2017-07-04 | 中国科学院深圳先进技术研究院 | A kind of method of real-time processing remote synchronization file |
| CN107301196A (en) * | 2017-05-03 | 2017-10-27 | 北京小度信息科技有限公司 | Method of data synchronization, device, equipment and monitoring server |
| CN108021674A (en) * | 2017-12-06 | 2018-05-11 | 浙江远算云计算有限公司 | A kind of multi-level buffer delivery acceleration system of synchronization high in the clouds emulation data |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN112035463A (en) * | 2020-07-22 | 2020-12-04 | 武汉达梦数据库有限公司 | Bidirectional synchronization method and synchronization device of heterogeneous database based on log analysis |
| US20210056124A1 (en) * | 2019-08-22 | 2021-02-25 | Adp, Llc | Robust Data Synchronization Solution Between Databases |
| CN112988897A (en) * | 2021-03-30 | 2021-06-18 | 工银科技有限公司 | Data bidirectional synchronization method and device in system upgrading scene |
-
2022
- 2022-06-16 CN CN202210676804.8A patent/CN114756628B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103823797A (en) * | 2012-11-16 | 2014-05-28 | 镇江诺尼基智能技术有限公司 | FTP (file transfer protocol) based real-time industry database data synchronization system |
| CN103839191A (en) * | 2014-02-21 | 2014-06-04 | 国家电网公司 | Method for integrally and synchronously maintaining custom-made model |
| CN103970833A (en) * | 2014-04-02 | 2014-08-06 | 浙江大学 | Method for achieving two-way synchronous data circulation in heterogeneous database synchronizing system based on logs |
| CN106919574A (en) * | 2015-12-24 | 2017-07-04 | 中国科学院深圳先进技术研究院 | A kind of method of real-time processing remote synchronization file |
| CN108628881A (en) * | 2017-03-20 | 2018-10-09 | 北京京东尚科信息技术有限公司 | Method of data synchronization and device |
| CN107301196A (en) * | 2017-05-03 | 2017-10-27 | 北京小度信息科技有限公司 | Method of data synchronization, device, equipment and monitoring server |
| CN108021674A (en) * | 2017-12-06 | 2018-05-11 | 浙江远算云计算有限公司 | A kind of multi-level buffer delivery acceleration system of synchronization high in the clouds emulation data |
| US20210056124A1 (en) * | 2019-08-22 | 2021-02-25 | Adp, Llc | Robust Data Synchronization Solution Between Databases |
| CN112035463A (en) * | 2020-07-22 | 2020-12-04 | 武汉达梦数据库有限公司 | Bidirectional synchronization method and synchronization device of heterogeneous database based on log analysis |
| CN112988897A (en) * | 2021-03-30 | 2021-06-18 | 工银科技有限公司 | Data bidirectional synchronization method and device in system upgrading scene |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114756628B (en) | 2022-10-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6873995B2 (en) | Method, system, and program product for transaction management in a distributed content management application | |
| US8504523B2 (en) | Database management system | |
| US7840539B2 (en) | Method and system for building a database from backup data images | |
| KR100983300B1 (en) | Recovery from failures within data processing systems | |
| US7472125B2 (en) | Method for managing a database system | |
| US8214377B2 (en) | Method, system, and program for managing groups of objects when there are different group types | |
| CN106570163B (en) | Audit log read-write management method and system facing unreliable environment | |
| EP0224681A2 (en) | Method for managing obsolescence of data objects | |
| US8756196B2 (en) | Propagating tables while preserving cyclic foreign key relationships | |
| US20090210429A1 (en) | System and method for asynchronous update of indexes in a distributed database | |
| US20190347167A1 (en) | Primary Node-Standby Node Data Transmission Method, Control Node, and Database System | |
| US8214376B1 (en) | Techniques for global single instance segment-based indexing for backup data | |
| KR20190063835A (en) | System for processing real-time data modification of in-memory database | |
| US20150006485A1 (en) | High Scalability Data Management Techniques for Representing, Editing, and Accessing Data | |
| US20180276267A1 (en) | Methods and system for efficiently performing eventual and transactional edits on distributed metadata in an object storage system | |
| CN114116717A (en) | Database information query method, device, equipment and readable storage medium | |
| CN114756628B (en) | High-multiplexing extensible data and file synchronization method | |
| CN111522688B (en) | Data backup method and device for distributed system | |
| CN116049306A (en) | Data synchronization method, device, electronic equipment and readable storage medium | |
| CN114564458B (en) | Method, device, equipment and storage medium for synchronizing data among clusters | |
| US7313603B2 (en) | System and method for synchronizing unstructured documents | |
| CN113672683B (en) | Spark SQL-based distributed database metadata synchronization device and method | |
| CN116821232A (en) | Data synchronization method and related device | |
| CN113076298A (en) | Distributed small file storage system | |
| CN107066348B (en) | Method and device for realizing high-availability NAS cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 300456 room 506-11, building 1-4, Rongsheng Plaza, Binhai Zhongguancun Science Park, Binhai New Area Economic and Technological Development Zone, Tianjin Patentee after: Tianjin 712 Mobile Communication Co.,Ltd. Country or region after: China Address before: 300456 room 506-11, building 1-4, Rongsheng Plaza, Binhai Zhongguancun Science Park, Binhai New Area Economic and Technological Development Zone, Tianjin Patentee before: TIANJIN 712 MOBILE COMMUNICATIONS Co.,Ltd. Country or region before: China |