CN117349384A - Database synchronization method, system and equipment - Google Patents
Database synchronization method, system and equipment Download PDFInfo
- Publication number
- CN117349384A CN117349384A CN202311639914.8A CN202311639914A CN117349384A CN 117349384 A CN117349384 A CN 117349384A CN 202311639914 A CN202311639914 A CN 202311639914A CN 117349384 A CN117349384 A CN 117349384A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- subsystem
- platform
- synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000012544 monitoring process Methods 0.000 claims abstract description 55
- 230000008859 change Effects 0.000 claims abstract description 25
- 238000004590 computer program Methods 0.000 claims description 10
- 230000015654 memory Effects 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 2
- 230000001360 synchronised effect Effects 0.000 abstract description 10
- 230000008569 process Effects 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
Abstract
The application discloses a method, a system and equipment for database synchronization, wherein a monitoring component is configured between databases to be synchronized, the databases are monitored in real time, when the occurrence of the change of the data in the databases is monitored, the changed data in the databases is obtained through the monitoring component, the monitoring component determines a target database to be synchronized according to the changed data and the changed databases, and meanwhile judges whether the target database comprises or partially comprises the data type of the changed data, and the changed data corresponding to the data type included in the target database is synchronized to the target database. The problem of data synchronization between a platform and a plurality of subsystem databases is solved.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to a method, a system, and an apparatus for database synchronization.
Background
With the continuous progress of electronic informatization of the whole society, more and more work adopts an electronic and informatization system.
Larger scale content or data service companies, governments, or medical facilities have complex organizations such as N municipal companies under province, M regional companies under each municipal company, and so on. The companies of each level need to have the same business work content and are independent and autonomous. Thus, each subsidiary will have a separate set of systems. Therefore, these superior companies often need to have a certain degree of data statistics capability, data analysis capability and data management capability, so that a design mode of a cascade system is formed.
At present, in the cascade system design, a plurality of bins are generally built and an ETL tool is configured for data synchronization, but the cost for building the plurality of bins is higher, and the main purpose is to perform data analysis, so that a data synchronization method of a cascade system is urgently needed under the scene that data analysis is not needed and only data synchronization is needed.
Disclosure of Invention
The invention provides a method, a system and equipment for synchronizing databases, and provides a low-cost cascade system data synchronization scheme which at least solves the problem of data synchronization between cascade systems.
The embodiment of the application provides a database synchronization method, which comprises the following steps:
monitoring a platform database and at least one subsystem database;
when the data of a first database in the platform database or at least one subsystem database is monitored to change, acquiring first data in the first database, wherein the first data is the data with the change in the first database;
when the first database is the platform database, a first subsystem database pointed by the first data is obtained according to the first data; obtaining second data matched with the first subsystem database according to the first data; and writing the second data into the first subsystem database;
when the first database is the subsystem database, third data matched with the platform database is obtained according to the first data; and writing the third data into the platform database.
Optionally, the step of monitoring the platform database and the at least one subsystem database includes:
and monitoring the platform database and at least one subsystem database through a cananal component, wherein the platform database and the subsystem database are databases supporting binary logs.
Optionally, the step of monitoring the platform database or the first database in at least one subsystem database for a change in data; the step of obtaining the first data in the first database comprises the following steps:
monitoring binary logs of the platform database and at least one subsystem database;
when the binary logs in a first database in the platform database and at least one subsystem database are monitored to change, the cananal component forwards the binary logs in the first database to a relay log of the cananal component;
and analyzing the data in the relay log to acquire the first data.
Optionally, when the first database is the platform database;
the first data comprises first tag data used for representing a target subsystem database;
the step of obtaining a first subsystem database pointed by the first data according to the first data comprises the following steps:
obtaining the first tag data in the first data according to the first data;
and according to the first tag data, a first subsystem database pointed by the first data is obtained.
Optionally, the step of obtaining second data matched with the first subsystem database according to the first data includes:
acquiring all data attributes in the first subsystem database according to the first subsystem database;
performing data matching on the first data according to all data attributes in the first subsystem database;
and obtaining the second data according to the result of data matching of the first data.
Optionally, the step of obtaining third data matched with the platform database according to the first data includes:
acquiring all data attributes in the platform database according to the platform database;
performing data matching on the first data according to all data attributes in the platform database;
and obtaining the third data according to the result of data matching of the first data.
Optionally, the method further comprises:
when the monitoring component monitors that the data in the first subsystem database changes before the step of writing the second data into the first subsystem database is completed;
acquiring fourth data in the first subsystem database through the monitoring component, wherein the fourth data is changed data in the first subsystem database;
judging whether the fourth data and the second data have the data with the same data attribute or not;
if there is data of the same data attribute, suspending writing the second data into the first subsystem database;
if the data with the same data attribute does not exist, writing the second data into the first subsystem database, and simultaneously obtaining fifth data matched with the platform database according to the fourth data;
writing the fifth data into the platform database;
when the monitoring component monitors that the data in the platform database changes before the step of writing the third data into the platform database is completed;
obtaining sixth data in the platform database through the monitoring component, wherein the sixth data is changed data in the platform database;
according to the sixth data, a second subsystem database pointed by the sixth data is obtained;
judging whether the second subsystem database and the first database are the same database or not;
if the data are the same database, judging whether the data with the same data attribute exist in the sixth data and the third data;
if the data with the same data attribute exists, stopping writing the third data into the platform database;
if the data with the same data attribute does not exist or the data with the same data attribute is not the same database, writing the third data into the platform database, and simultaneously obtaining seventh data matched with the second subsystem database according to the sixth data;
writing the seventh data to the second subsystem database.
In yet another aspect, embodiments of the present application further provide a system for database synchronization, including a data synchronization device, a platform database, and at least one subsystem database, wherein:
the data synchronization device is configured to:
monitoring the platform database and at least one subsystem database;
when the data of a first database in the platform database or at least one subsystem database is monitored to change, acquiring first data in the first database, wherein the first data is the data with the change in the first database;
when the first database is the platform database, a first subsystem database pointed by the first data is obtained according to the first data; obtaining second data matched with the first subsystem database according to the first data; and writing the second data into the first subsystem database;
when the first database is the subsystem database, third data matched with the platform database is obtained according to the first data; writing the third data into the platform database;
the platform database is configured to:
receiving input of a user, and modifying data in the platform database according to the input of the user;
responding to the request of the monitoring component, and transmitting the data with the change in the platform database to the data synchronization equipment;
receiving input of the data synchronization equipment, and modifying data in the platform database according to the input of the data synchronization equipment;
the subsystem database is configured to:
receiving user input, and modifying data in the subsystem database according to the user input;
transmitting data with changes in the subsystem database to the data synchronization device in response to the request of the monitoring component;
and receiving the input of the data synchronization device, and modifying the data in the subsystem database according to the input of the data synchronization device.
Optionally, the data in the platform database and the subsystem database have at least partially identical data attributes.
In yet another aspect, embodiments of the present application further provide a computer device including a memory and a processor, where the memory stores a computer program, and the processor executes the computer program to implement the above method.
In yet another aspect, embodiments of the present application further provide a computer readable storage medium having a computer program stored thereon, and a processor executing the computer program to implement the above method.
Compared with the prior art, the invention has the following advantages and beneficial effects:
the invention relates to a method, a system and equipment for synchronizing databases, wherein a monitoring component is configured among databases needing to be synchronized, the databases are monitored in real time, when the occurrence of the change of the data in the databases is monitored, the changed data in the databases is obtained through the monitoring component, the monitoring component determines a target database needing to synchronize the data according to the changed data and the changed data, and meanwhile, whether the target database comprises or partially comprises the changed data type is judged, and the changed data corresponding to the data type included in the target database is synchronized to the target database. The problem of data synchronization between a platform and a plurality of subsystem databases is solved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following description will make brief description of the drawings used in the description of the embodiments or the prior art. Like elements or portions are generally identified by like reference numerals throughout the several figures. In the drawings, elements or portions thereof are not necessarily drawn to scale.
FIG. 1 is a flow chart of a method of database synchronization in the present application;
fig. 2 is a schematic diagram of a database synchronization system in the present application.
The realization, functional characteristics and advantages of the present application will be further described with reference to the embodiments, referring to the attached drawings.
Detailed Description
In order that those skilled in the art will better understand the present disclosure, a clear and complete description of the technical solutions of the embodiments of the present disclosure will be provided below in conjunction with the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only some embodiments of the present disclosure, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without inventive effort, based on the embodiments in this disclosure, shall fall within the scope of the present disclosure.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
As shown in fig. 1, a method for synchronizing databases includes a data synchronizing device, a platform database, and at least one subsystem database, and is executed by the data synchronizing device, where the data synchronizing device is a computer device having data processing and data storage capabilities, and further includes the following steps:
s1, monitoring a platform database and at least one subsystem database;
the data synchronization device is provided with a monitoring component for monitoring the platform database and the at least one subsystem database, and the monitoring component is configured to acquire changed data from a first database in the platform database or the at least one subsystem database as first data when the data of the first database changes;
optionally, the step of monitoring the platform database and the at least one subsystem database includes:
and monitoring the platform database and at least one subsystem database by the cananal assembly, wherein the platform database and the subsystem database are databases supporting binary logs.
Specifically, the listening component is configured as a journal component, and the platform database and the subsystem database are databases supporting binary logs. Specifically, the platform database and the subsystem database are MySQL databases.
The MySQL database itself supports master (host) slave (slave) backups, and its principle is:
the Master host sends the change record to a binary file (binary log);
the Slave sends dump protocol from the library (Slave) to the Master, copies binary log events of the Master into its relay log (relay log);
slave reads and redos events in the relay log from the library, synchronizing the changed data to its own database.
The working principle of synchronizing by using the cananal component is as follows:
the cap component simulates an interactive protocol of MySQL Slave, simulates the Slave to be a Slave library, and sends a dump protocol to the Master library;
the Master library receives the dump request and starts pushing the binary log to the Slave library (the cap component);
the cap component parses the binary log object to obtain data modification records in the Master database to determine the changed data in the database.
S2, when the change of the data of a first database in the platform database or at least one subsystem database is monitored, acquiring the first data in the first database;
when the monitoring component monitors that the data of the first database in the platform database or the at least one subsystem database changes;
the data synchronization device acquires first data in a first database through a monitoring component, wherein the first data is changed data in the first database;
the data synchronization device monitors the platform database or at least one subsystem database in real time through the monitoring component, when the data in a certain database in the platform database or at least one subsystem database is monitored to change, the database is used as a first database, and the changed data in the database is used as the first data.
Optionally, when the monitoring component monitors that the data of the first database in the platform database or the at least one subsystem database changes;
the method comprises the steps of obtaining first data in a first database through a monitoring component, wherein the first data are changed data in the first database, and the steps comprise:
the monitoring component monitors the binary logs of the platform database and at least one subsystem database;
when the change of the binary logs in the first database in the platform database and the at least one subsystem database is monitored, the monitoring component simulates a slave of the first database and transfers the binary logs in the first database to a relay log of the monitoring component;
the monitoring component parses the data in the relay log to obtain first data.
Specifically, the binary log records all DDL and DML statements in the form of events. Mainly for backup and data synchronization. There are three types of binary logs: STATEMENT, ROW, MIXED;
STATEMENT: recorded are SQL statements of execution;
ROW: the record is a real data record;
MIXED ed: recorded is STATEMENT +ROW, preferably in STATEMENT mode.
S3, judging the type of the first database;
when the first database is a platform database;
s4, according to the first data, a first subsystem database pointed by the first data is obtained; obtaining second data matched with the first subsystem database according to the first data; writing the second data into the first subsystem database;
when the first database is a subsystem database;
s5, obtaining third data matched with the platform database according to the first data; and writing the third data to the platform database.
Because one platform database may correspond to a plurality of subsystem databases, and data between the subsystem databases is not needed or cannot be communicated, the platform database needs to directionally synchronize the first data to the target subsystem database, namely the first subsystem database, so as to avoid the problems of data leakage and the like caused by synchronizing the first data to other databases, and therefore, when the first database is the platform database, the first subsystem database needs to be determined.
According to the scheme, the monitoring components are configured between the databases to be synchronized, the databases are monitored in real time, when the data in the databases are monitored to change, the monitoring components acquire the changed data in the databases, the monitoring components determine the target databases to be synchronized according to the changed data and the changed data, and judge whether the target databases comprise or partially comprise the changed data types, and synchronize the changed data corresponding to the data types in the target databases to the target databases. The problem of data synchronization between a platform and a plurality of subsystem databases is solved. Compared with a method for constructing a plurality of bins, the scheme is convenient to construct, metadata is not required to be maintained, and compared with a scheme of regular synchronization, the scheme is better in instantaneity. Compared with the existing configuration scheme of the cananal assembly, the configuration scheme is capable of configuring the monitoring assembly, meanwhile, the data synchronization equipment where the monitoring assembly is located can also process the monitored changed data, a target database needing to be synchronized is determined, screening is conducted, and preprocessing is conducted on the data needing to be synchronized, so that the scheme can be better adapted to a cascade system.
Optionally, when the first database is a platform database;
the first data comprises first tag data used for representing a target subsystem database;
the step of obtaining a first subsystem database pointed by the first data according to the first data comprises the following steps:
the data synchronization device obtains first tag data in the first data according to the first data;
and the data synchronization device obtains a first subsystem database pointed by the first data according to the first tag data.
By adopting the method, the data synchronization device can read the first tag data which are pointed to the target database in the first data, the first tag data can be determined by special characters, special positions or other preset means, the data content of the first tag data can be the physical address, IP address or number and the like of the target database, and the data synchronization device acquires the target database which needs to synchronize the first data through the first tag data.
Optionally, the step of obtaining second data matched with the first subsystem database according to the first data includes:
the data synchronization device obtains all data attributes in the first subsystem database according to the first subsystem database;
the data synchronization device performs data matching on the first data according to all data attributes in the first subsystem database;
and the data synchronization device obtains second data according to the result of data matching of the first data.
Optionally, when the first database is a subsystem database, the first tag data is empty or points to the platform database.
Optionally, the step of obtaining third data matched with the platform database according to the first data includes:
the data synchronization device obtains all data attributes in the platform database according to the platform database;
the data synchronization device performs data matching on the first data according to all data attributes in the platform database;
and the data synchronization device obtains third data according to the result of data matching of the first data.
Because the data synchronization is carried out by adopting the update-monitoring-synchronization method, an ETL tool is not used, other bin counting technologies are not used, the cost is effectively controlled, the threshold of an access platform is reduced, and meanwhile, some problems are brought, namely, the data attributes in a data link table are not completely the same between a platform database and a subsystem database, between the subsystem database and the subsystem database, for example, the data attributes of the platform database can comprise: name, gender, time of job entry, compensation, scheduling, and data attributes in the subsystem database include: when the data synchronization is carried out, the platform database only needs to synchronize the data related to the name, the sex and the scheduling to the subsystem database, and the subsystem database only needs to synchronize the data related to the name, the sex and the scheduling to the platform database.
Optionally, the method further comprises:
when the monitoring component monitors that the data in the first subsystem database changes before the step of writing the second data into the first subsystem database is completed;
acquiring fourth data in the first subsystem database through the monitoring component, wherein the fourth data is changed data in the first subsystem database;
judging whether the fourth data and the second data have the data with the same data attribute or not;
if there is data with the same data attribute, stopping writing the second data into the first subsystem database;
if the data with the same data attribute does not exist, writing the second data into the first subsystem database, and simultaneously obtaining fifth data matched with the platform database according to the fourth data;
writing fifth data into a platform database;
when the monitoring component monitors that the data in the platform database changes before the step of writing the third data into the platform database is completed;
acquiring sixth data in the platform database through the monitoring component, wherein the sixth data is changed data in the platform database;
according to the sixth data, a second subsystem database pointed by the sixth data is obtained;
judging whether the second subsystem database is the same database as the first database;
if the data are the same database, judging whether the data with the same data attribute exist in the sixth data and the third data;
if the data with the same data attribute exists, stopping writing the third data into the platform database;
if the data with the same data attribute does not exist or the data with the same data attribute is not the same database, writing the third data into the platform database, and simultaneously obtaining seventh data matched with the second subsystem database according to the sixth data;
and writing seventh data into the second subsystem database.
In the use process, because the platform database manager and the subsystem database manager can receive the data to be updated at the same time, after the platform database manager and the subsystem database manager update the data of the platform database and the subsystem database at the same time, when the platform database synchronizes the data of the subsystem database, the corresponding subsystem database synchronizes the data of the platform database, for example, the platform database manager and the subsystem database manager receive the information of the personnel "first" needed to be newly added in the database and write the information of the personnel "first" into the platform database and the subsystem database, the platform database writes the information of the personnel "first" into the platform database and distributes UID1 for the first, the subsystem database writes the information of the first "into the subsystem database and distributes UID2 for the first, and then the platform database and the subsystem database synchronize the information of the first to each other respectively, if the method is not adopted, after the platform database receives the synchronization of the subsystem database, the same information of the 'A' is written into the subsystem database and then distributed with a UID3 for the 'A', after the subsystem database receives the synchronization of the platform database, the same information of the 'A' is written into the subsystem database and then distributed with a UID4 for the 'A', because the corresponding information of the UID1 and the UID3, the UID2 and the UID4 are the same, various problems are easy to occur when the data of the 'A' are queried, managed and analyzed subsequently, therefore, when the platform database and the subsystem database are synchronized with the data of the other side simultaneously or respectively in a certain time interval, the data synchronization equipment stops the synchronization and notifies the manager of the corresponding database, and the next operation is carried out according to the input of the manager, the next operation comprises but is not limited to stop the synchronization of the two sides, terminating synchronization of one of the parties and preserving synchronization of both parties.
Optionally, encrypted communications are used between the platform database and the data synchronization device, and between the subsystem database and the data synchronization device.
Optionally, the first data, the second data, the third data, the fourth data, the fifth data, the sixth data and the seventh data are all stored with a backup in the data synchronization device, and the backup can be rolled, circularly stored or deleted according to the requirement of the user.
Example 2
As shown in fig. 2, a system for database synchronization includes a data synchronization device, a platform database, and at least one subsystem database, optionally, the number of subsystem databases is 3, where:
the data synchronization device is configured to:
monitoring a platform database and at least one subsystem database;
when the change of the data of a first database in the platform database or at least one subsystem database is monitored, acquiring the first data in the first database, wherein the first data is the data with the change in the first database;
when the first database is a platform database, acquiring a first subsystem database pointed by the first data according to the first data; obtaining second data matched with the first subsystem database according to the first data; writing the second data into the first subsystem database;
when the first database is a subsystem database, third data matched with the platform database is obtained according to the first data; writing third data into a platform database;
the platform database is configured to:
receiving user input, and modifying data in a platform database according to the user input;
responding to the request of the monitoring component, and transmitting the data with the change in the platform database to the data synchronization equipment;
receiving input of a data synchronization device, and modifying data in a platform database according to the input of the data synchronization device;
the subsystem database is configured to:
receiving user input, and modifying data in a subsystem database according to the user input;
responding to the request of the monitoring component, and transmitting the data with the change in the subsystem database to the data synchronization device;
and receiving input of the data synchronization device, and modifying data in the subsystem database according to the input of the data synchronization device.
Optionally, the data synchronization device monitors the platform database and the at least one subsystem database by setting a cap component.
Alternatively, the platform database may be a subsystem database of a higher level database, in which case the higher level system serves as the platform database.
Optionally, the data in the platform database and the subsystem database have at least partially identical data attributes.
Data attributes such as a platform database may include: name, gender, time of job entry, compensation, scheduling, and data attributes in the subsystem database include: when the data synchronization is carried out, the platform database only needs to synchronize the data related to the name, the sex and the scheduling to the subsystem database.
Example 3
The present embodiment provides a computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement any of the methods described above.
Example 4
The present embodiment provides a computer-readable storage medium, on which a computer program is stored, and a processor executes the computer program to implement any one of the methods described above.
In some embodiments, the computer readable storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; but may be a variety of devices including one or any combination of the above memories. The computer may be a variety of computing devices including smart terminals and servers.
In the foregoing embodiments of the present disclosure, the descriptions of the various embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed technology content may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of units may be a logic function division, and there may be another division manner in actual implementation, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable non-volatile storage medium. Based on such understanding, the technical solution of the present disclosure may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a non-volatile storage medium, including several instructions to cause a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of the various embodiments of the present disclosure. And the aforementioned nonvolatile storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely a preferred embodiment of the present disclosure and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present disclosure and are intended to be comprehended within the scope of the present disclosure.
Claims (10)
1. A method of database synchronization, comprising the steps of:
monitoring a platform database and at least one subsystem database;
when the data of a first database in the platform database or at least one subsystem database is monitored to change, acquiring first data in the first database, wherein the first data is the data with the change in the first database;
when the first database is the platform database, a first subsystem database pointed by the first data is obtained according to the first data; obtaining second data matched with the first subsystem database according to the first data; and writing the second data into the first subsystem database;
when the first database is the subsystem database, third data matched with the platform database is obtained according to the first data; and writing the third data into the platform database.
2. The method of database synchronization of claim 1, wherein the step of listening to the platform database and the at least one subsystem database comprises:
and monitoring the platform database and at least one subsystem database through a cananal component, wherein the platform database and the subsystem database are databases supporting binary logs.
3. A method of database synchronization according to claim 2, wherein said detecting when a change in data of a first database of said platform database or at least one of said sub-system databases occurs; the step of obtaining the first data in the first database comprises the following steps:
monitoring binary logs of the platform database and at least one subsystem database;
when the binary logs in a first database in the platform database and at least one subsystem database are monitored to change, the cananal component forwards the binary logs in the first database to a relay log of the cananal component;
and analyzing the data in the relay log to acquire the first data.
4. A method of database synchronization according to claim 1, wherein when the first database is the platform database;
the first data comprises first tag data used for representing a target subsystem database;
the step of obtaining a first subsystem database pointed by the first data according to the first data comprises the following steps:
obtaining the first tag data in the first data according to the first data;
and according to the first tag data, a first subsystem database pointed by the first data is obtained.
5. A method of database synchronization according to claim 1, wherein said step of obtaining second data matching said first subsystem database from said first data comprises:
acquiring all data attributes in the first subsystem database according to the first subsystem database;
performing data matching on the first data according to all data attributes in the first subsystem database;
and obtaining the second data according to the result of data matching of the first data.
6. A method of database synchronization according to claim 1, wherein said step of obtaining third data matching said platform database from said first data comprises:
acquiring all data attributes in the platform database according to the platform database;
performing data matching on the first data according to all data attributes in the platform database;
and obtaining the third data according to the result of data matching of the first data.
7. A method of database synchronization according to claim 1, further comprising:
when the monitoring component monitors that the data in the first subsystem database changes before the step of writing the second data into the first subsystem database is completed;
acquiring fourth data in the first subsystem database through the monitoring component, wherein the fourth data is changed data in the first subsystem database;
judging whether the fourth data and the second data have the data with the same data attribute or not;
if there is data of the same data attribute, suspending writing the second data into the first subsystem database;
if the data with the same data attribute does not exist, writing the second data into the first subsystem database, and simultaneously obtaining fifth data matched with the platform database according to the fourth data;
writing the fifth data into the platform database;
when the monitoring component monitors that the data in the platform database changes before the step of writing the third data into the platform database is completed;
obtaining sixth data in the platform database through the monitoring component, wherein the sixth data is changed data in the platform database;
according to the sixth data, a second subsystem database pointed by the sixth data is obtained;
judging whether the second subsystem database and the first database are the same database or not;
if the data are the same database, judging whether the data with the same data attribute exist in the sixth data and the third data;
if the data with the same data attribute exists, stopping writing the third data into the platform database;
if the data with the same data attribute does not exist or the data with the same data attribute is not the same database, writing the third data into the platform database, and simultaneously obtaining seventh data matched with the second subsystem database according to the sixth data;
writing the seventh data to the second subsystem database.
8. A system for database synchronization comprising a data synchronization device, a platform database, and at least one subsystem database, wherein:
the data synchronization device is configured to:
monitoring the platform database and at least one subsystem database;
when the data of a first database in the platform database or at least one subsystem database is monitored to change, acquiring first data in the first database, wherein the first data is the data with the change in the first database;
when the first database is the platform database, a first subsystem database pointed by the first data is obtained according to the first data; obtaining second data matched with the first subsystem database according to the first data; and writing the second data into the first subsystem database;
when the first database is the subsystem database, third data matched with the platform database is obtained according to the first data; writing the third data into the platform database;
the platform database is configured to:
receiving input of a user, and modifying data in the platform database according to the input of the user;
responding to the request of the monitoring component, and transmitting the data with the change in the platform database to the data synchronization equipment;
receiving input of the data synchronization equipment, and modifying data in the platform database according to the input of the data synchronization equipment;
the subsystem database is configured to:
receiving user input, and modifying data in the subsystem database according to the user input;
transmitting data with changes in the subsystem database to the data synchronization device in response to the request of the monitoring component;
and receiving the input of the data synchronization device, and modifying the data in the subsystem database according to the input of the data synchronization device.
9. The system for database synchronization of claim 8, wherein the data in the platform database and the subsystem database have at least partially identical data attributes.
10. A computer device, characterized in that it comprises a memory in which a computer program is stored and a processor which executes the computer program, implementing the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311639914.8A CN117349384B (en) | 2023-12-04 | 2023-12-04 | Database synchronization method, system and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311639914.8A CN117349384B (en) | 2023-12-04 | 2023-12-04 | Database synchronization method, system and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117349384A true CN117349384A (en) | 2024-01-05 |
| CN117349384B CN117349384B (en) | 2024-03-15 |
Family
ID=89356007
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311639914.8A Active CN117349384B (en) | 2023-12-04 | 2023-12-04 | Database synchronization method, system and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117349384B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104679888A (en) * | 2015-03-17 | 2015-06-03 | 浪潮集团有限公司 | Device and method for data synchronization |
| US20180246886A1 (en) * | 2017-02-27 | 2018-08-30 | OSF Global Services Inc. | Data migration for platform integration |
| CN108920659A (en) * | 2018-07-03 | 2018-11-30 | 广州唯品会信息科技有限公司 | Data processing system and its data processing method, computer readable storage medium |
| CN109739929A (en) * | 2018-12-18 | 2019-05-10 | 中国人民财产保险股份有限公司 | Method of data synchronization, apparatus and system |
| CN110245181A (en) * | 2019-04-25 | 2019-09-17 | 深圳壹账通智能科技有限公司 | Data synchronous synchronous method, device and computer storage medium |
| CN110309156A (en) * | 2018-03-01 | 2019-10-08 | 阿里巴巴集团控股有限公司 | Database Systems, database update, expansion method and equipment |
| US20210056119A1 (en) * | 2019-08-22 | 2021-02-25 | Sap Se | Automated performing of replication tasks in a multiple database system |
| CN113297166A (en) * | 2020-07-27 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Data processing system, method and device |
| CN115544172A (en) * | 2022-11-28 | 2022-12-30 | 杭州欧若数网科技有限公司 | Method and system for synchronizing data among clusters of one master and multiple slaves in real time |
| CN115705343A (en) * | 2021-08-09 | 2023-02-17 | 上海宝信软件股份有限公司 | Data synchronization method and system for master node and slave node of industrial internet platform |
| CN116108109A (en) * | 2023-02-28 | 2023-05-12 | 北京江民新科技术有限公司 | Cascade data synchronization method, system, equipment and storage medium based on Canal |
| CN117041269A (en) * | 2023-08-08 | 2023-11-10 | 北京金信融通科技有限公司 | Real-time bidirectional synchronization system for synchronizing sub-database data |
-
2023
- 2023-12-04 CN CN202311639914.8A patent/CN117349384B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104679888A (en) * | 2015-03-17 | 2015-06-03 | 浪潮集团有限公司 | Device and method for data synchronization |
| US20180246886A1 (en) * | 2017-02-27 | 2018-08-30 | OSF Global Services Inc. | Data migration for platform integration |
| CN110309156A (en) * | 2018-03-01 | 2019-10-08 | 阿里巴巴集团控股有限公司 | Database Systems, database update, expansion method and equipment |
| CN108920659A (en) * | 2018-07-03 | 2018-11-30 | 广州唯品会信息科技有限公司 | Data processing system and its data processing method, computer readable storage medium |
| CN109739929A (en) * | 2018-12-18 | 2019-05-10 | 中国人民财产保险股份有限公司 | Method of data synchronization, apparatus and system |
| CN110245181A (en) * | 2019-04-25 | 2019-09-17 | 深圳壹账通智能科技有限公司 | Data synchronous synchronous method, device and computer storage medium |
| US20210056119A1 (en) * | 2019-08-22 | 2021-02-25 | Sap Se | Automated performing of replication tasks in a multiple database system |
| CN113297166A (en) * | 2020-07-27 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Data processing system, method and device |
| CN115705343A (en) * | 2021-08-09 | 2023-02-17 | 上海宝信软件股份有限公司 | Data synchronization method and system for master node and slave node of industrial internet platform |
| CN115544172A (en) * | 2022-11-28 | 2022-12-30 | 杭州欧若数网科技有限公司 | Method and system for synchronizing data among clusters of one master and multiple slaves in real time |
| CN116108109A (en) * | 2023-02-28 | 2023-05-12 | 北京江民新科技术有限公司 | Cascade data synchronization method, system, equipment and storage medium based on Canal |
| CN117041269A (en) * | 2023-08-08 | 2023-11-10 | 北京金信融通科技有限公司 | Real-time bidirectional synchronization system for synchronizing sub-database data |
Non-Patent Citations (2)
| Title |
|---|
| QIZHI ZHANG ET AL.: "A survey on the efficiency, reliability, and security of data query in blockchain systems", 《FUTURE GENERATION COMPUTER SYSTEMS》, pages 303 - 320 * |
| 申利民 等: "面向协同系统集成的数据同步模型", 《计算机应用研究》, pages 1384 - 1386 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117349384B (en) | 2024-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7090744B2 (en) | Distributed database cluster system and data synchronization method | |
| CN107402963B (en) | Search data construction method, incremental data pushing device and equipment | |
| US10726042B2 (en) | Replication control using eventually consistent meta-data | |
| CN102265277A (en) | Operation method and device for data memory system | |
| CN111177165B (en) | Method, device and equipment for detecting data consistency | |
| CN107977396B (en) | Method and device for updating data table of KeyValue database | |
| CN102833281B (en) | It is a kind of distributed from the implementation method counted up, apparatus and system | |
| US10976942B2 (en) | Versioning a configuration of data storage equipment | |
| CN104657497A (en) | Mass electricity information concurrent computation system and method based on distributed computation | |
| CN109298978B (en) | Recovery method and system for database cluster of specified position | |
| CN103488793A (en) | User behavior monitoring method based on information retrieval | |
| JP2020057416A (en) | Method and device for processing data blocks in distributed database | |
| CN114048217A (en) | Incremental data synchronization method and device, electronic equipment and storage medium | |
| CN107040576A (en) | Information-pushing method and device, communication system | |
| CN112559475A (en) | Data real-time capturing and transmitting method and system | |
| CN114416868A (en) | Data synchronization method, device, equipment and storage medium | |
| CN106682141B (en) | Data synchronization method based on service operation log | |
| CN117349384B (en) | Database synchronization method, system and equipment | |
| CN116521083A (en) | Storage method, device and medium for distributed storage cluster data | |
| CN114238324A (en) | Checking method and device for host station, electronic equipment and storage medium | |
| CN103778218A (en) | Cloud computation-based standard information consistency early warning system and method | |
| CN110837454A (en) | Interface monitoring processing method and server | |
| CN106375354B (en) | Data processing method and device | |
| CN113609199B (en) | Database system, server, and storage medium | |
| CN115766498A (en) | Big data link full link tracking monitoring method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |