CN114328747A - Data synchronization method, data synchronization device, computer equipment and medium - Google Patents
Data synchronization method, data synchronization device, computer equipment and medium Download PDFInfo
- Publication number
- CN114328747A CN114328747A CN202111671128.7A CN202111671128A CN114328747A CN 114328747 A CN114328747 A CN 114328747A CN 202111671128 A CN202111671128 A CN 202111671128A CN 114328747 A CN114328747 A CN 114328747A
- Authority
- CN
- China
- Prior art keywords
- target
- data
- data records
- program
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The present disclosure relates to a data synchronization method, apparatus, computer device and medium; wherein, the method comprises the following steps: acquiring data to be synchronized from a source database; packaging data to be synchronized into a plurality of data records, and sequentially writing the data records into corresponding target queues; and calling a target data reading program to read the corresponding data record from the target queue and sending the read data record to the corresponding target database. According to the embodiment of the disclosure, the corresponding data records are read from the target queue, so that the occupation of source database resources can be reduced, and the real-time performance of data synchronization of the target database is ensured.
Description
Technical Field
The present disclosure relates to the field of data synchronization technologies, and in particular, to a data synchronization method, apparatus, computer device, and medium.
Background
With the continuous development of computer network technology, database synchronization technology is becoming a key technology for ensuring system performance and improving system reliability. When the data synchronization tool is used for data synchronization, the data synchronization is mainly divided into three stages: the first stage is to carry out initialization loading of stock data to obtain a basic point of data synchronization; in the second stage, incremental data synchronization is carried out by taking a basic point obtained by loading the initialized data as a reference; and the third stage is used for regularly comparing and checking the source data and the target data of the data synchronization so as to confirm that no data is lost in the data synchronization process.
In a synchronization scenario of data distribution, especially in the second stage of incremental data synchronization, incremental data of one source database needs to be synchronized to multiple target databases at the same time. In a conventional data synchronization method, a plurality of data capture programs are installed in a source database, and an original data distribution scene is converted into a plurality of 1-to-1 data synchronization scenes. However, in the above method, because a plurality of data capture programs all need to capture incremental data from the source database, a large amount of Central Processing Unit (CPU) resources and Input/output (I/O) resources of the source database may be occupied, and for the source database, the load is large and a large amount of resources are wasted.
Disclosure of Invention
In order to solve the technical problem, the present disclosure provides a data synchronization method, apparatus, computer device and medium.
In a first aspect, the present disclosure provides a data synchronization method, including:
acquiring data to be synchronized from a source database;
packaging the data to be synchronized into a plurality of data records, and sequentially writing the data records into corresponding target queues;
and calling a target data reading program to read the corresponding data record from the target queue and sending the read data record to the corresponding target database.
Optionally, the invoking a target data reading program reads the corresponding data record from the target queue, and sends the read data record to the corresponding target database, including:
determining a corresponding target data reading program according to the target database;
acquiring a data reading mode;
and calling the target data reading program to read the corresponding target data record from the target queue according to the data reading mode, and sending the target data record to the target database.
Optionally, the sending the target data record to the target database includes:
determining a target warehousing program corresponding to the target database;
and sending the target data record to the target warehousing program so that the target warehousing program sends the target data record to the target database.
Optionally, the encapsulating the data to be synchronized into a plurality of data records, and sequentially writing the plurality of data records into corresponding target queues includes:
calling a corresponding data writing program to package the data to be synchronized into a plurality of data records, and determining a first number of all data records contained in the plurality of data records;
acquiring a second number of storage intervals in the target queue;
and sequentially writing the plurality of data records into corresponding target queues according to the size relationship between the first number and the second number.
Optionally, the sequentially writing the plurality of data records into corresponding target queues according to the size relationship between the first number and the second number includes:
if the first number is larger than the second number, sequentially writing a target number of data records into corresponding storage intervals based on the arrangement sequence of the data records in the plurality of data records, wherein the target number is equal to the second number;
and determining that an idle storage interval exists in the target queue, and sequentially writing a third number of data records into the corresponding idle storage interval according to a third number of the idle storage interval and the arrangement sequence corresponding to the remaining data records in the plurality of data records.
Optionally, the method further includes:
and if the first number is less than or equal to the second number, sequentially writing the data records into the corresponding storage sections based on the arrangement sequence of the data records in the data records.
Optionally, the determining that a free storage interval exists in the target queue includes:
acquiring the times of respectively reading the data records corresponding to each storage interval in the target queue;
and determining that an idle storage interval exists in the target queue according to all times.
In a second aspect, the present disclosure provides a data synchronization apparatus, including:
the acquisition module is used for acquiring data to be synchronized from a source database;
the write-in module is used for packaging the data to be synchronized into a plurality of data records and sequentially writing the data records into corresponding target queues;
and the sending module is used for calling a target data reading program to read the corresponding data record from the target queue and sending the read data record to the corresponding target database.
Optionally, the sending module includes:
the first determining unit is used for determining a corresponding target data reading program according to the target database;
the second determining unit is used for acquiring a data reading mode;
and the record sending unit is used for calling the target data reading program to read the corresponding target data record from the target queue according to the data reading mode and sending the target data record to the target database.
Optionally, the record sending unit is specifically configured to:
calling the target data reading program to read the corresponding target data record from the target queue according to the data reading mode;
determining a target warehousing program corresponding to the target database;
and sending the target data record to the target warehousing program so that the target warehousing program sends the target data record to the target database.
Optionally, the writing module includes:
the packaging unit is used for calling a corresponding data writing program to package the data to be synchronized into a plurality of data records and determining a first number of all data records contained in the plurality of data records;
the number acquisition unit is used for acquiring a second number of storage intervals in the target queue;
and the record writing unit is used for sequentially writing the plurality of data records into the corresponding target queues according to the size relationship between the first number and the second number.
Optionally, the recording and writing unit includes:
a first writing unit, configured to, if the first number is greater than the second number, sequentially write a target number of data records into a corresponding storage interval based on an arrangement order of the data records in the plurality of data records, where the target number is equal to the second number;
and the second writing unit is used for determining that an idle storage interval exists in the target queue and sequentially writing a third number of data records into the corresponding idle storage interval according to a third number of the idle storage interval and the arrangement sequence corresponding to the rest of the data records in the plurality of data records.
Optionally, the recording and writing unit is specifically configured to:
and if the first number is less than or equal to the second number, sequentially writing the data records into the corresponding storage sections based on the arrangement sequence of the data records in the data records.
Optionally, the second writing unit is specifically configured to:
acquiring the times of respectively reading the data records corresponding to each storage interval in the target queue;
determining that an idle storage interval exists in the target queue according to all times;
and sequentially writing the third number of data records into the corresponding idle storage intervals according to the third number of the idle storage intervals and the arrangement sequence corresponding to the rest data records in the plurality of data records.
In a third aspect, the present disclosure also provides a computer device, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the data synchronization method of any of the embodiments of the present disclosure.
In a fourth aspect, the present disclosure also provides a computer-readable storage medium on which a computer program is stored, which when executed by a processor, implements the data synchronization method described in any one of the embodiments of the present disclosure.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages: the method comprises the steps of firstly obtaining data to be synchronized from a source database, then packaging the data to be synchronized into a plurality of data records, sequentially writing the data records into corresponding target queues, finally calling a target data reading program to read the corresponding data records from the target queues, sending the read data records to the corresponding target databases, and reading the corresponding data records from the target queues, so that the occupation of source database resources can be reduced, and meanwhile, the real-time performance of data synchronization of the target databases is guaranteed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic view of an application scenario of a data synchronization method provided in an embodiment of the present disclosure;
fig. 2A is a schematic flow chart of a data synchronization method provided by an embodiment of the present disclosure;
FIG. 2B is a schematic diagram of a data synchronization process in an embodiment of the disclosure;
fig. 3A is a schematic flow chart of another data synchronization method provided by the embodiment of the present disclosure;
fig. 3B is a schematic diagram of an interaction process between a source database, a data capture program, and a target database provided by an embodiment of the present disclosure;
FIG. 3C is a schematic diagram illustrating an implementation of a target queue provided by an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of a data synchronization apparatus provided in an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
Fig. 1 is a schematic view of an application scenario of a data synchronization method provided in an embodiment of the present disclosure, as shown in fig. 1:
in a synchronization scenario of data distribution, data of one source-end database needs to be synchronized to multiple target-end databases at the same time, that is: and synchronizing the data of the source database to the target database 1, the target database 2, the target database 3 and the target database 4 at the same time. In the traditional method, 4 data capture programs (the number of which is consistent with the number of target databases) are installed in a source database, and an original data distribution scene is converted into 4 1-to-1 data synchronization scenes, but the method can consume a large amount of resources of the source database, cause a large burden on the source database, and influence the performance of the source database.
It should be noted that: the number of the target databases in fig. 1 is an exemplary illustration, and may be other numbers, and the present embodiment does not specifically limit the number, performance, name, and other attributes of the target databases.
Fig. 2A is a schematic flowchart of a data synchronization method according to an embodiment of the present disclosure. The embodiment can be applied to the situation that the data of one source database is synchronized to a plurality of target databases at the same time. The method of the embodiment may be performed by a data synchronization apparatus, which may be implemented in hardware and/or software and may be configured in a computer device. As shown in fig. 2A, the method specifically includes the following steps:
s210, obtaining the data to be synchronized from the source database.
Wherein, the source database can be understood as a source database that needs to synchronize data to other databases. The data to be synchronized may be understood as data that needs to be synchronized to other databases in the source database, and may include stock data or incremental data of the source database, and the like.
In order to solve the problems in the prior art, in this embodiment, a data capture program is deployed in a source database, and data that needs to be synchronized to other databases in the source database is captured by the data capture program, so that data to be synchronized can be obtained.
In some embodiments, data that needs to be synchronized in the source database may be captured specifically by an analysis program in the data capture program, for example, an analysis thread or an analysis process, or may be captured by other manners, which is not limited in this embodiment.
S220, packaging the data to be synchronized into a plurality of data records, and writing the data records into corresponding target queues in sequence.
Wherein a package may be understood as a split. A data record may be understood as a complete set of related information corresponding to a line of information in the data to be synchronized. A target queue may be understood as a memory queue for storing data records.
After the data to be synchronized is acquired by the data capture program, as the data to be synchronized is possibly more, in order to improve the subsequent synchronization efficiency, the data to be synchronized can be packaged into one data record through the data capture program to obtain a plurality of data records, and the plurality of data records are sequentially written into the corresponding target queues according to the splitting sequence.
In some embodiments, the encapsulation and the data recording writing may be performed by a data writing program in the data capture program. The data writer may include a write thread or a write process, and the embodiment is not limited in particular.
In the embodiment, only one data capture program is deployed, the data to be synchronized is packaged into a plurality of data records, and the plurality of data records are sequentially written into the corresponding target queues, so that the data to be synchronized is ensured to be acquired from the source database only once, and the consumption of resources in the source database can be reduced.
And S230, calling a target data reading program to read the corresponding data record from the target queue, and sending the read data record to the corresponding target database.
The target data reading program may be understood as a program capable of reading data records from the target queue and sending the read data records to the corresponding target database, such as a read thread or a read process. The target data reading program corresponds to the target database one by one, namely: each target database has a corresponding target data reading procedure. A target database may be understood as a database that receives data to be synchronized. The number of the target databases may be multiple, and this embodiment is not particularly limited. The data capture program can comprise a plurality of target data reading programs, and the number of the target data reading programs is the same as that of the target databases.
After the data capture program writes the plurality of data records into the corresponding target queues in sequence, the corresponding data records can be read from the target queues by calling a target data reading program in the data capture program, and after the data records are read, the read data records are sent to the corresponding target databases. Because the number of the target databases is consistent with that of the target data reading programs, under the condition that a plurality of target databases exist, the data records can be simultaneously read from the target queue through the target data reading programs respectively corresponding to the target databases, and therefore the real-time performance of the target databases in the data synchronization process is guaranteed.
In this embodiment, first, data to be synchronized is obtained from a source database, then, the data to be synchronized is encapsulated into a plurality of data records, the plurality of data records are sequentially written into corresponding target queues, finally, a target data reading program is called to read corresponding data records from the target queues, the read data records are sent to corresponding target databases, and by reading corresponding data records from the target queues, occupation of source database resources can be reduced, and real-time performance of data synchronization of the target databases is guaranteed.
In this embodiment, optionally, the acquiring the data to be synchronized from the source database may specifically include:
acquiring a database log corresponding to the source database;
and analyzing the database log to obtain the data to be synchronized.
Specifically, when data in the source database is written in, added, deleted, and changed, the data is synchronized to a database log corresponding to the source database, so that by obtaining the database log corresponding to the source database and analyzing the database log, writing in, adding, deleting, and changing conditions of the data in the source database can be obtained, and thus data to be synchronized is obtained.
In the embodiment, the data to be synchronized is obtained by the method, so that the method is simple and efficient, and errors are not easy to occur.
In this embodiment, optionally, the invoking the target data reading program reads the corresponding data record from the target queue, and sends the read data record to the corresponding target database, which may specifically include:
determining a corresponding target data reading program according to the target database;
acquiring a data reading mode;
and calling the target data reading program to read the corresponding target data record from the target queue according to the data reading mode, and sending the target data record to the target database.
Specifically, after the data capture program sequentially writes the plurality of data records into the corresponding target queues, the data records in the target queues need to be synchronized to the corresponding target databases, in the synchronization process, a target data reading program corresponding to the target database needs to be determined according to the target database, which may be determined according to a corresponding relationship between the target database and the target data reading program, or according to identification information of the target database, which is not specifically limited in this embodiment. Secondly, the data reading mode to be obtained can be determined according to the requirements of the target database, for example, some target databases need to obtain all target data records in the target queue, and some target databases need to filter the target data records contained in the target queue to obtain part of the target data records; the data reading mode may also be obtained by other methods, and this embodiment is not limited. And finally, calling a target data reading program to read the target data record corresponding to the data reading mode from the target queue and sending the target data record to a corresponding target database.
For example, assuming that the target database 1 corresponds to the target data reading program 1, and the target database 1 is to filter the target data records contained in the target queue and only acquire the data records related to the a table, when synchronizing the data to be synchronized to the target database 1, the target data reading program 1 is invoked to read the target data records related to the a table from the target queue and send the target data records related to the a table to the target database 1, so that the target database 1 can receive the required data.
In this embodiment, by the above method, smooth progress of the data synchronization process can be ensured, data synchronization efficiency is improved, and errors are prevented.
In this embodiment, optionally, the sending the target data record to the target database may specifically include:
determining a target warehousing program corresponding to the target database;
and sending the target data record to the target warehousing program so that the target warehousing program sends the target data record to the target database.
The target entering program may be understood as a data entering program corresponding to the target database, and the target entering program also corresponds to a certain target data reading program included in the data capturing program.
Specifically, since the target warehousing program corresponds to both the target database and a certain target data reading program included in the data capture program, after the target data reading program is called to read a corresponding target data record from the target queue, the target warehousing program corresponding to the target database needs to be determined, and then the target data record is sent to the target warehousing program, so that the target warehousing program can send the target data record to the corresponding target database.
In this embodiment, through the correspondence between the target storage program, the target database, and the target data reading program, an error in the target data record during the synchronization process can be prevented, and the accuracy of data synchronization is improved.
For example, fig. 2B is a schematic diagram of a data synchronization process in an embodiment of the disclosure. Fig. 2B includes 1 source database, 1 data capture program, 4 target warehousing programs, and 4 target databases, where the target warehousing program 1 corresponds to the target database 1, the target warehousing program 2 corresponds to the target database 2, the target warehousing program 3 corresponds to the target database 3, and the target warehousing program 4 corresponds to the target database 4. In the data synchronization process, the data capture program is responsible for acquiring data to be synchronized from a source database, packaging the data to be synchronized into a plurality of data records, sequentially writing the data records into corresponding target queues, calling a target data reading program to read the corresponding data records from the target queues, and sending the read data records to corresponding target warehousing programs, so that the target warehousing programs send the read data records to target databases corresponding to the target warehousing programs.
It should be noted that the number of the target warehousing programs and the target databases in fig. 2B is used to describe the data synchronization process, and this embodiment does not specifically limit this.
Fig. 3A is a schematic flowchart of another data synchronization method provided in the embodiment of the present disclosure. The embodiment is optimized on the basis of the embodiment. Optionally, this embodiment explains in detail a process of encapsulating data to be synchronized into a plurality of data records and writing the plurality of data records into corresponding target queues in sequence. As shown in fig. 3A, the method specifically includes the following steps:
s310, obtaining the data to be synchronized from the source database.
S320, calling the corresponding data writing program to package the data to be synchronized into a plurality of data records, and determining the first number of all the data records contained in the plurality of data records.
After the data capture program acquires the data to be synchronized, the data to be synchronized can be packaged into a plurality of data records by calling the corresponding data write-in program, and meanwhile, after the packaging is completed, the total number of all the data records, namely the first number, can be determined by counting the number of all the data records contained in the plurality of data records.
S330, acquiring a second number of storage intervals in the target queue.
Specifically, the target queue is located in the memory, and the storage interval of the target queue is limited, so that the second number of the storage intervals included in the target queue needs to be acquired in order to facilitate writing of subsequent data records.
And S340, sequentially writing the plurality of data records into the corresponding target queues according to the magnitude relation of the first number and the second number.
After the first number and the second number are obtained, the sizes of the first number and the second number are compared, and according to the size relationship of the first number and the second number, whether the plurality of data records are sequentially written into the corresponding target queues at one time or the plurality of data records are sequentially written into the corresponding target queues in batches can be determined.
And S350, calling a target data reading program to read the corresponding data record from the target queue and sending the read data record to the corresponding target database.
In this embodiment, first, data to be synchronized is obtained from a source database, then a corresponding data writing program is called to encapsulate the data to be synchronized into a plurality of data records, a first number of all data records included in the plurality of data records is determined, a second number of storage intervals in a target queue is obtained, then the plurality of data records are sequentially written into the corresponding target queue according to a magnitude relationship between the first number and the second number, finally a target data reading program is called to read the corresponding data records from the target queue, the read data records are sent to the corresponding target database, by comparing the first number of all data records included in the plurality of data records with the second number of storage intervals in the target queue, a writing mode for sequentially writing the plurality of data records into the corresponding target queue can be determined, and the method is more flexible and has stronger applicability, meanwhile, the corresponding data records are read from the target queue, so that the occupation of source database resources can be reduced, and the real-time performance of data synchronization of the target database is ensured.
In this embodiment, optionally, the sequentially writing the plurality of data records into the corresponding target queues according to the size relationship between the first number and the second number may specifically include:
if the first number is larger than the second number, sequentially writing a target number of data records into corresponding storage intervals based on the arrangement sequence of the data records in the plurality of data records, wherein the target number is equal to the second number;
and determining that an idle storage interval exists in the target queue, and sequentially writing a third number of data records into the corresponding idle storage interval according to a third number of the idle storage interval and the arrangement sequence corresponding to the remaining data records in the plurality of data records.
The free storage interval may be understood as a storage interval in which a data record is located if the data record is read by all target data reading programs, and the data record can be overwritten by a new data record.
Specifically, if the first number is greater than the second number, it indicates that the total number of the data records is greater than the number of the storage sections, at this time, the data records are required to be written into the corresponding storage sections in batches, first, the data records of the target number are obtained according to the arrangement sequence of each data record in the plurality of data records, that is, the first N (N is equal to the target number) data records are obtained from the plurality of data records, then, the first N data records are sequentially written into the corresponding storage sections according to the corresponding arrangement sequence, and finally, when it is determined that a free storage section exists in the target queue, the data records of the third number are obtained from the remaining data records and are sequentially written into the corresponding free storage sections according to the third number of the free storage sections and the arrangement sequence corresponding to the remaining data records in the plurality of data records.
In this embodiment, when the first number is greater than the second number, the data records are written by the above method, so that the storage interval can be reasonably planned, and the writing efficiency is improved.
In this embodiment, optionally, the method further includes:
and if the first number is less than or equal to the second number, sequentially writing the data records into the corresponding storage sections based on the arrangement sequence of the data records in the data records.
Specifically, if the first number is equal to or less than the second number, it indicates that the total number of data records is equal to or less than the number of storage sections, and in this case, the data records can be directly written into the corresponding storage sections, and the plurality of data records can be sequentially written into the corresponding storage sections according to the arrangement order of the data records in the plurality of data records during writing.
In this embodiment, when the first number is greater than the second number, the data record is written by the above method, which saves time and makes full use of all storage sections.
In this embodiment, optionally, the determining that the idle storage interval exists in the target queue may specifically include:
acquiring the times of respectively reading the data records corresponding to each storage interval in the target queue;
and determining that an idle storage interval exists in the target queue according to all times.
Specifically, the number of times that the data records corresponding to each storage interval in the target queue are respectively read by different target data reading programs is obtained, and if the number of times that at least 1 storage interval in the target queue is read is equal to the number of target data reading programs (i.e., the number of target databases), it is indicated that a free storage interval exists in the target queue.
In this embodiment, the method determines that the idle storage interval exists in the target queue, and the data records included in the subsequent remaining data records can be written into the idle storage interval conveniently, so that the writing efficiency is improved.
Fig. 3B is a schematic diagram of an interaction process between a source database, a data capture program, and a target database provided by an embodiment of the present disclosure, and as shown in fig. 3B, an implementation manner is exemplarily shown.
The data capture program includes an analysis program, a data writing program, a target queue, 4 target data reading programs (target data reading program 1, target data reading program 2, target data reading program 3, and target data reading program 4), 4 target warehousing programs (target warehousing program 1, target warehousing program 2, target warehousing program 3, and target warehousing program 4), and 4 target databases (target database 1, target database 2, target database 3, and target database 4), and the functions of the programs have been described in the above embodiments, and are not described herein again. The target data reading program, the target warehousing program and the target database have corresponding relations shown in the figure, the target data reading program 1, the target warehousing program 1 and the target database 1 correspond to each other, and …, the target data reading program 4, the target warehousing program 4 and the target database 4 correspond to each other. The object queue includes n (n is a positive integer) storage sections, and stores data record 1, data record 2, data record 3, data records 4 and …, and data record n, respectively. The progress of reading the data record by each target data reading program can be different, and the target data reading programs do not interfere with each other in the reading process.
In fig. 3B, the data capture program is a process, and the rest of the parsing program, the data writing program, and the 4 target data reading programs are threads, that is: the method comprises the steps of analyzing threads, writing threads, reading threads 1, reading threads 2, reading threads 3 and reading threads 4, so that a thread framework with one writing thread and a plurality of reading threads is formed, the writing threads and the plurality of reading threads of the framework are not interfered with each other, and a source end database does not need to write local files, so that IO load is not generated.
In some embodiments, the parser, the data writer, and the 4 target data readers may also be processes, i.e., a parser process, a write process, a read process 1, a read process 2, a read process 3, and a read process 4, thereby constituting a process architecture with one write process and multiple read processes. In this case, shared memory may be used for storing the target queue.
It should be noted that: the number of target data reading programs, the number of target warehousing programs and the number of target databases are not limited in this embodiment.
For example, fig. 3C is a schematic diagram of an implementation manner of the target queue provided by the embodiment of the present disclosure, as shown in fig. 3C:
a) dividing a memory queue into M (M is a positive integer) storage intervals for storing M data records (the M data records are ordered, and the storage intervals can be stored circularly);
b) setting an array of global variables, each element (number) of the array marking the current reading condition of the data record in 1 storage interval, and the initial value of each element in the array is 0, for example, element 0 marks the current reading condition of data record 1, …, and element 6 marks the current reading condition of data record 7;
c) reading data records 1 to 7 by each target data reading program respectively; when reading a certain data record each time, adding one to an element corresponding to the data record in the array;
d) when writing data record, the data writing program sequentially writes from the first storage interval to the next storage interval;
i. checking whether the array element value corresponding to each storage interval is equal to the number of the target data reading programs, if so, indicating that all the target data reading programs read the data record, directly writing a new data record, covering the data record, and changing the corresponding array element value to 0 to indicate that the new data record has not been read by the target data reading programs;
if the array element value corresponding to a certain storage interval is smaller than the number of the target data reading programs, the target data reading programs do not read the data records, and waiting is carried out until the array element value is equal to the number of the target data reading programs.
It should be noted that: the number of elements, the number of storage sections, and the number of data records included in the array are not limited in this embodiment.
Fig. 4 is a schematic structural diagram of a data synchronization apparatus provided in an embodiment of the present disclosure; the device is configured in computer equipment, and can realize the data synchronization method in any embodiment of the application. The device specifically comprises the following steps:
an obtaining module 410, configured to obtain data to be synchronized from a source database;
a write module 420, configured to encapsulate the data to be synchronized into a plurality of data records, and write the plurality of data records into corresponding target queues in sequence;
and a sending module 430, configured to invoke a target data reading program to read a corresponding data record from the target queue, and send the read data record to a corresponding target database.
In this embodiment, optionally, the sending module 430 includes:
the first determining unit is used for determining a corresponding target data reading program according to the target database;
the second determining unit is used for acquiring a data reading mode;
and the record sending unit is used for calling the target data reading program to read the corresponding target data record from the target queue according to the data reading mode and sending the target data record to the target database.
In this embodiment, optionally, the record sending unit is specifically configured to:
calling the target data reading program to read the corresponding target data record from the target queue according to the data reading mode;
determining a target warehousing program corresponding to the target database;
and sending the target data record to the target warehousing program so that the target warehousing program sends the target data record to the target database.
In this embodiment, optionally, the writing module 420 includes:
the packaging unit is used for calling a corresponding data writing program to package the data to be synchronized into a plurality of data records and determining a first number of all data records contained in the plurality of data records;
the number acquisition unit is used for acquiring a second number of storage intervals in the target queue;
and the record writing unit is used for sequentially writing the plurality of data records into the corresponding target queues according to the size relationship between the first number and the second number.
In this embodiment, optionally, the recording and writing unit includes:
a first writing unit, configured to, if the first number is greater than the second number, sequentially write a target number of data records into a corresponding storage interval based on an arrangement order of the data records in the plurality of data records, where the target number is equal to the second number;
and the second writing unit is used for determining that an idle storage interval exists in the target queue and sequentially writing a third number of data records into the corresponding idle storage interval according to a third number of the idle storage interval and the arrangement sequence corresponding to the rest of the data records in the plurality of data records.
In this embodiment, optionally, the recording and writing unit is specifically configured to:
and if the first number is less than or equal to the second number, sequentially writing the data records into the corresponding storage sections based on the arrangement sequence of the data records in the data records.
In this embodiment, optionally, the second writing unit is specifically configured to:
acquiring the times of respectively reading the data records corresponding to each storage interval in the target queue;
determining that an idle storage interval exists in the target queue according to all times;
and sequentially writing the third number of data records into the corresponding idle storage intervals according to the third number of the idle storage intervals and the arrangement sequence corresponding to the rest data records in the plurality of data records.
According to the data synchronization device provided by the embodiment of the disclosure, data to be synchronized is firstly acquired from a source database, then the data to be synchronized is packaged into a plurality of data records, the data records are sequentially written into corresponding target queues, finally a target data reading program is called to read the corresponding data records from the target queues, the read data records are sent to the corresponding target databases, and the corresponding data records are read from the target queues, so that the occupation of source database resources can be reduced, and meanwhile, the real-time performance of data synchronization of the target databases is ensured.
The data synchronization device provided by the embodiment of the disclosure can execute the data synchronization method provided by any embodiment of the disclosure, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure. As shown in fig. 5, the computer apparatus includes a processor 510 and a storage device 520; the number of the processors 510 in the computer device may be one or more, and one processor 510 is taken as an example in fig. 5; the processor 510 and the storage 520 in the computer device may be connected by a bus or other means, as exemplified by the bus connection in fig. 5.
The storage device 520, which is a computer-readable storage medium, can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the data synchronization method in the embodiments of the present disclosure. The processor 510 executes various functional applications and data processing of the computer device by executing software programs, instructions and modules stored in the storage device 520, namely, implements the data synchronization method provided by the embodiment of the present disclosure.
The storage device 520 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the storage 520 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the storage 520 may further include memory located remotely from the processor 510, which may be connected to a computer device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The computer device provided by this embodiment can be used to execute the data synchronization method provided by any of the above embodiments, and has corresponding functions and advantages.
The disclosed embodiments also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are used to implement the data synchronization method provided by the disclosed embodiments.
Of course, the storage medium provided by the embodiments of the present disclosure contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the data synchronization method provided by any embodiments of the present disclosure.
From the above description of the embodiments, it is obvious for a person skilled in the art that the present disclosure can be implemented by software and necessary general hardware, and certainly can be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present disclosure may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present disclosure.
It should be noted that, in the embodiment of the data synchronization apparatus, the included units and modules are only divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only used for distinguishing one functional unit from another, and are not used for limiting the protection scope of the present disclosure.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method for synchronizing data, the method comprising:
acquiring data to be synchronized from a source database;
packaging the data to be synchronized into a plurality of data records, and sequentially writing the data records into corresponding target queues;
and calling a target data reading program to read the corresponding data record from the target queue and sending the read data record to the corresponding target database.
2. The method of claim 1, wherein the invoking a target data reading program reads corresponding data records from the target queue and sends the read data records to a corresponding target database, comprising:
determining a corresponding target data reading program according to the target database;
acquiring a data reading mode;
and calling the target data reading program to read the corresponding target data record from the target queue according to the data reading mode, and sending the target data record to the target database.
3. The method of claim 2, wherein sending the target data record to the target database comprises:
determining a target warehousing program corresponding to the target database;
and sending the target data record to the target warehousing program so that the target warehousing program sends the target data record to the target database.
4. The method according to claim 1, wherein said encapsulating the data to be synchronized into a plurality of data records and writing the plurality of data records into corresponding target queues in sequence comprises:
calling a corresponding data writing program to package the data to be synchronized into a plurality of data records, and determining a first number of all data records contained in the plurality of data records;
acquiring a second number of storage intervals in the target queue;
and sequentially writing the plurality of data records into corresponding target queues according to the size relationship between the first number and the second number.
5. The method of claim 4, wherein sequentially writing the plurality of data records into corresponding target queues according to the magnitude relationship between the first number and the second number comprises:
if the first number is larger than the second number, sequentially writing a target number of data records into corresponding storage intervals based on the arrangement sequence of the data records in the plurality of data records, wherein the target number is equal to the second number;
and determining that an idle storage interval exists in the target queue, and sequentially writing a third number of data records into the corresponding idle storage interval according to a third number of the idle storage interval and the arrangement sequence corresponding to the remaining data records in the plurality of data records.
6. The method of claim 4, further comprising:
and if the first number is less than or equal to the second number, sequentially writing the data records into the corresponding storage sections based on the arrangement sequence of the data records in the data records.
7. The method of claim 5, wherein the determining that there is a free storage interval in the target queue comprises:
acquiring the times of respectively reading the data records corresponding to each storage interval in the target queue;
and determining that an idle storage interval exists in the target queue according to all times.
8. A data synchronization apparatus, the apparatus comprising:
the acquisition module is used for acquiring data to be synchronized from a source database;
the write-in module is used for packaging the data to be synchronized into a plurality of data records and sequentially writing the data records into corresponding target queues;
and the sending module is used for calling a target data reading program to read the corresponding data record from the target queue and sending the read data record to the corresponding target database.
9. A computer device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111671128.7A CN114328747A (en) | 2021-12-31 | 2021-12-31 | Data synchronization method, data synchronization device, computer equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111671128.7A CN114328747A (en) | 2021-12-31 | 2021-12-31 | Data synchronization method, data synchronization device, computer equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114328747A true CN114328747A (en) | 2022-04-12 |
Family
ID=81020971
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111671128.7A Pending CN114328747A (en) | 2021-12-31 | 2021-12-31 | Data synchronization method, data synchronization device, computer equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114328747A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116860869A (en) * | 2023-05-29 | 2023-10-10 | 玖章算术(浙江)科技有限公司 | Queue delivery method and system under primary key concurrency scene |
-
2021
- 2021-12-31 CN CN202111671128.7A patent/CN114328747A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116860869A (en) * | 2023-05-29 | 2023-10-10 | 玖章算术(浙江)科技有限公司 | Queue delivery method and system under primary key concurrency scene |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110888727B (en) | Method, device and storage medium for realizing concurrent lock-free queue | |
| CN108459962B (en) | Code normalization detection method and device, terminal equipment and storage medium | |
| CN112486913B (en) | Log asynchronous storage method and device based on cluster environment | |
| CN114595199B (en) | File analysis method and device, computer equipment and storage medium | |
| CN114328747A (en) | Data synchronization method, data synchronization device, computer equipment and medium | |
| CN111240772A (en) | Data processing method and device based on block chain and storage medium | |
| CN112965939A (en) | File merging method, device and equipment | |
| CN110795920B (en) | Document generation method and device | |
| CN112000589A (en) | Data writing method, data reading device and electronic equipment | |
| US20180033115A1 (en) | Capturing Commands in a Multi-Engine Graphics Processing Unit | |
| JP2018132948A (en) | Loading program, loading method, and information processing device | |
| CN108073709B (en) | Data recording operation method, device, equipment and storage medium | |
| US9659041B2 (en) | Model for capturing audit trail data with reduced probability of loss of critical data | |
| CN114265906A (en) | Data synchronization method, data synchronization device, computer equipment and medium | |
| CN112764673B (en) | Hyperspectral linear array data storage rate optimization method, device and storage medium | |
| CN110716923B (en) | Data processing method, data processing device, node equipment and storage medium | |
| CN108388424B (en) | Method and device for calling interface data and electronic equipment | |
| CN113312386A (en) | Batch warehousing method based on distributed messages | |
| CN114830103A (en) | Tracking handler module system and method of using the same | |
| CN113568578B (en) | Picture processing method and device, electronic equipment and readable storage medium | |
| CN112035471A (en) | Transaction processing method and computer equipment | |
| CN111651531A (en) | Data import method, device, equipment and computer storage medium | |
| CN114925026B (en) | Electronic file utilization method and device, storage medium and electronic equipment | |
| CN112558975B (en) | Method and device for switching codes of components, electronic equipment and readable storage medium | |
| CN113835989A (en) | Performance statistical method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |