CN110674214B - Big data synchronization method, device, computer equipment and storage medium - Google Patents
Big data synchronization method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110674214B CN110674214B CN201910841880.8A CN201910841880A CN110674214B CN 110674214 B CN110674214 B CN 110674214B CN 201910841880 A CN201910841880 A CN 201910841880A CN 110674214 B CN110674214 B CN 110674214B
- Authority
- CN
- China
- Prior art keywords
- database
- data
- history
- synchronization
- historical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 230000001360 synchronised effect Effects 0.000 claims abstract description 51
- 238000012360 testing method Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 9
- 238000011010 flushing procedure Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000002354 daily effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000003442 weekly effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a big data synchronization method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: responding to a synchronous data acquisition instruction sent by a local terminal, and acquiring a synchronous completion state of a local historical database; wherein the history database is used for executing batch synchronization operation on first history data generated by a remote system during a first reference period; selecting a local first database or a local second database as a target incremental database according to the synchronization completion state of the historical database; wherein the first database is used for synchronizing the first historical data generated by the remote system during the first reference in real time, and the second database is used for synchronizing the second historical data generated by the remote system during the second reference in real time; taking all data in the history database and the target incremental database as target synchronous data; and sending the target synchronous data to the local terminal.
Description
Technical Field
The present invention relates to the field of data synchronization technologies, and in particular, to a method and apparatus for synchronizing big data, a computer device, and a storage medium.
Background
From the perspective of a big data development platform, for reasons of component technical architecture or service security isolation, various operations or search processing cannot be directly performed on data stored or generated by an online service system, so that the data needs to be collected into various storage computing components of the development platform for processing. In this process, data synchronization becomes a very important link.
In the data synchronization service, a large number of data synchronization tasks are to synchronize data of the previous day in the vicinity of early morning to wait for batch analysis. The following problems may often be encountered in such a scenario: the data may not be synchronized due to too large an amount of information, e.g., when the data synchronization task on the first day begins to execute, the data synchronization task on the second day has not yet been completed. In this case, the operation of the data of the previous day cannot be queried in time, for example, the insurance industry queries for data of a certain insurance order generated on the previous day.
Therefore, how to solve the problem that the synchronization process takes a long time and cannot acquire the newly added data in time is a technical problem to be solved by the technicians in the field.
Disclosure of Invention
The invention aims to provide a big data synchronization method, a big data synchronization device, computer equipment and a storage medium, so as to solve the problems in the prior art.
In order to achieve the above object, the present invention provides a big data synchronization method, comprising the steps of:
responding to a synchronous data acquisition instruction sent by a local terminal, and acquiring a synchronous completion state of a local historical database; wherein the history database is used for executing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for characterizing whether the history database has completed batch synchronization of the first history data;
selecting a local first database or a local second database as a target incremental database according to the synchronization completion state of the historical database; wherein the first database is used for synchronizing the first historical data generated by the remote system in real time during the first reference period, and the second database is used for synchronizing the second historical data generated by the remote system in real time during a second reference period, wherein the second reference period is the next time period arranged continuously with the first reference period;
taking all data in the history database and the target incremental database as target synchronous data;
and sending the target synchronous data to the local terminal.
According to the big data synchronization method provided by the invention, the step of selecting the first database or the second database as the target incremental database according to the synchronization completion state of the history database comprises the following steps:
if the history database completes batch synchronization of the first history data, selecting the second database as a target incremental database;
and if the history database is not completely synchronized with the first history data, selecting the first database as a target incremental database.
According to the big data synchronization method provided by the invention, after the step of selecting the second database as the target incremental database, the method further comprises the following steps:
the data in the first database is emptied and third historical data generated by a remote system during a third reference period is synchronized to the first database in real time, wherein the third reference period is the next time period arranged in succession with the second reference period.
According to the big data synchronization method provided by the invention, after the first database is selected as the target incremental database, the method further comprises the following steps:
and (c) emptying data in the second database and synchronizing third historical data generated by a remote system during a third reference period to the second database in real time, wherein the third reference period is the next time period arranged in series with the second reference period.
According to the big data synchronization method provided by the invention, the step of acquiring the synchronization completion state of the local historical database comprises the following steps:
inquiring test data from the historical database, wherein the test data comprises a time field corresponding to the first reference period;
if the test data can be queried from the historical database, the historical database completes batch synchronization of the historical data;
and if the test data cannot be queried from the historical database, completing batch synchronization of the historical data by the historical database.
In order to achieve the above object, the present invention further provides a big data synchronization device, including:
the state acquisition module is adapted to respond to a synchronous data acquisition instruction sent by the local terminal and acquire the synchronous completion state of the local historical database; wherein the history database is used for executing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for characterizing whether the history database has completed batch synchronization of the first history data;
the increment selection module is suitable for selecting a local first database or a local second database as a target increment database according to the synchronous completion state of the historical database; wherein the first database is used for synchronizing the first historical data generated by the remote system in real time during the first reference period, and the second database is used for synchronizing the second historical data generated by the remote system in real time during a second reference period, wherein the second reference period is the next time period arranged continuously with the first reference period;
the data determining module is suitable for taking all data in the history database and the target incremental database as target synchronous data;
and the data transmitting module is suitable for transmitting the target synchronous data to the local terminal.
According to the big data synchronizing device provided by the invention, the target increment selection module comprises:
the first increment selection module is suitable for selecting the second database as a target increment database if the history database completes batch synchronization of the first history data;
and the second increment selection module is suitable for selecting the first database as a target increment database if the history database is not completely synchronized with the first history data.
The big data synchronizing device provided by the invention is characterized by further comprising:
a first flushing module adapted to flush data in the first database after selecting the second database as a target delta database and synchronize third historical data generated by a remote system during a third reference period to the first database in real time, wherein the third reference period is a next time period arranged in series with the second reference period;
and the second emptying module is suitable for emptying data in the second database after the first database is selected as a target incremental database, and synchronizing third historical data generated by a remote system during a third reference to the second database in real time.
To achieve the above object, the present invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the computer program.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above method.
The big data synchronization method, the big data synchronization device, the computer equipment and the storage medium provide a rapid and accurate data synchronization scheme for mass data. The invention synchronizes the first historical data generated by the remote system in the first reference period in batches through the historical database, and simultaneously synchronizes the first historical data and the second historical data generated in the first reference period and the second reference period in real time through the first database and the second database, wherein the second reference period is the next time period which is arranged continuously with the first reference period. In consideration of the time delay of the data synchronization process, the method and the device correspondingly select the first data and the second data as the target incremental data according to different synchronization completion conditions of the historical data, so that the historical data and the target incremental data form repeated-free total data together, and reliable execution of the data synchronization is ensured.
Drawings
FIG. 1 is a flowchart of a big data synchronization method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a program module of a big data synchronization apparatus according to an embodiment of the present invention;
fig. 3 is a schematic hardware structure of a big data synchronization device according to a first embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The big data synchronization method, the big data synchronization device, the computer equipment and the storage medium provide a rapid and accurate data synchronization scheme for mass data in a remote system. The invention synchronizes the first historical data generated by the remote system in the first reference period in batches through the historical database, and simultaneously synchronizes the first historical data and the second historical data generated in the first reference period and the second reference period in real time through the first database and the second database, wherein the second reference period is the next time period which is arranged continuously with the first reference period. In consideration of the time delay of the data synchronization process, the method and the device correspondingly select the first data and the second data as the target incremental data according to different synchronization completion conditions of the historical data, so that the historical data and the target incremental data form repeated-free total data together, and reliable execution of the data synchronization is ensured.
Example 1
Referring to fig. 1, the present embodiment provides a big data synchronization method, which specifically includes the following steps:
s1, responding to a synchronous data acquisition instruction sent by a local terminal, and acquiring a synchronous completion state of a local historical database; wherein the history database is used for executing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion status is used to characterize whether the history database has completed batch synchronization of the first history data.
The invention is applicable to any situation where it is desired to synchronize data in a remote system. For example, local police departments synchronize national id card information in national security systems, local companies of insurance companies need to synchronize customer policy data in national security systems, and so on. Taking an insurance company as an example, each local branch company can comprise a plurality of terminal devices, when any one terminal device wants to view national customer policy data, the national insurance system is not required to be directly connected, and the corresponding national customer policy data is acquired through a local server by sending a synchronous data acquisition instruction to the local server.
In response to a synchronization data acquisition instruction sent by the terminal device, the local server first needs to acquire a synchronization completion state of the local history database. Wherein the history database is used for executing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion status is used to characterize whether the history database has completed batch synchronization of the first history data.
The first reference period in the present invention is not a fixed specific period of time, but a sequence of periods of time with a regular arrangement feature, the first reference time may be set to be daily, weekly, monthly, etc., and typically the task start time of the first reference time is a fixed point of time, for example, zero each day, 20 nights each weekday; 00, 23:00 evening on the last day of each month. The history database in the present invention follows the T-1 pattern when synchronizing data in batches, i.e. the data-synchronization task during the previous first reference period starts to be executed at the beginning of each synchronization task, e.g. the data-batch synchronization task of the previous day is executed every night zero. Taking customer policy data in an insurance company as an example, the history data in the remote system is synchronized to a local history database, which is equivalent to synchronizing all customer policy data generated in the national insurance system from zero point every day.
It should be noted that, the history database in the present invention performs a batch data synchronization operation, that is, for batch data generated in a certain period of time, only after all data in the batch data are executed, operations such as querying, reading, modifying, etc. may be performed on the batch data in the history database. For example, the history database performs batch synchronization on data generated by a remote system on 1 month and 1 day from 1 month and 2 days, the whole synchronization process takes 3 hours, and the synchronization work on the 1 month and 1 day data is completed at 1 month and 2 days and 3 points, so that from 1 month and 2 days and 3 points, a user can only inquire the 1 month and 1 day data in the history database, and before 1 month and 2 days and 3 points, the user cannot inquire the 1 month and 1 day data. In summary, for the first historical data that is not completed in the process of executing the batch data synchronization, it cannot be queried in the historical database; the first history data may be queried in the history database only after the batch synchronization is completed.
For the above reasons, the present invention needs to acquire whether the history database has completed the batch synchronization operation on the first history data in order to determine whether the first history data can be queried from the history database.
S2, selecting a local first database or a local second database as a target incremental database according to the synchronous completion state of the historical database; wherein the first database is used for synchronizing the first historical data generated by the remote system in real time during the first reference period, and the second database is used for synchronizing the second historical data generated by the remote system in real time during a second reference period, wherein the second reference period is the next time period which is arranged continuously with the first reference period.
Because the batch data synchronization cannot inquire the synchronized first historical data before all the batch data synchronization is performed, in order to ensure that all the data in the remote system can be acquired in real time, the invention is provided with two local incremental databases, namely a first database and a second database, which are respectively used for storing the incremental data generated by the remote system from different moments. Specifically, the first database is used for synchronizing first historical data generated by the remote system in real time during the first reference period, and the second database is used for synchronizing second historical data generated by the remote system in real time during a second reference period, wherein the second reference period is the next time period arranged continuously with the first reference period. For example, if the first reference period is 1 month and 1 day, the second reference period is 1 month and 2 days; if the first reference period is 1 month for the first week, the second reference period is 1 month for the second week; if the first reference period is 2019, 1 month, the second reference period is 2019, 2 months, and so on. The first database and the second database adopt a real-time data synchronization mode, namely any data change in the remote database can be timely reflected to the first database or the second database, and the time occupied by executing the data synchronization operation is negligible. This is because the interval of real-time data synchronization is short, for example, it is typically synchronized once in ten minutes, or one half hour. In such a short interval, the amount of new data generated by the remote system is small, so that the synchronization operation is performed for a small number of synchronization times that are much shorter than for batch synchronization, and negligible.
Those skilled in the art can understand that, because real-time synchronization needs to constantly detect the data change condition in the remote system at any time, the occupied memory resources are more, but the real-time synchronization method has the advantages of good real-time performance and capability of timely acquiring the data change condition in the remote system. Trading off data integrity by sacrificing part of the memory resources is a major advantage for the present invention.
At this time, the invention includes three local databases, namely, a history database, a first database and a second database. Assuming that the first reference period is calculated in units of days and the start time is set to zero daily, a history database for performing batch synchronization is set as database a, a first database for performing real-time synchronization is set as database B, and a second database for performing real-time synchronization is set as database C. Assuming that database A needs to perform batch synchronization on data for 1 month 22, the present invention will synchronize data generated by the remote system for 1 month 22 in real time via database B and data generated by the remote system for 1 month 23 in real time via database C.
Still taking database a, database B and database C above as examples. If the current time is 1 month and 23 days, then 1 month and 22 days of data should be synchronized in database A. If the database a at the present time has completed synchronization, all the history data of 1 month 22 day and before are stored in the database a, and then the data generated from 1 month 23 is added to form complete full-volume data. According to the arrangement of the invention, the data generated from day 1, 23 is recorded in the database C, so that the database C now becomes the target incremental database.
Considering another case, if the database a at the current time has not completed the synchronization task, the database a does not include data of 1 month and 22 days, but only data of 1 month and 21 days and before. In this case, the data generated from day 22 of 1 month is added to form the complete data. According to the arrangement of the invention, data generated from day 1, 22 is recorded in database B, so that database B now becomes the target incremental database.
Regarding completion of batch synchronization of first historical data of the remote system by the historical database, the present invention determines by:
inquiring test data from the historical database, wherein the test data comprises a time field corresponding to the first reference time;
if the test data can be queried from the history database, the history database completes synchronization of the history data; and if the test data cannot be queried from the historical database, the historical database is not completely synchronized with the historical data.
And S3, taking all data in the history database and the target incremental database as target synchronous data.
On the basis that the database A has completed the current synchronization task, the data in the database A and the database C form complete full data, namely target synchronization data. When the order information of a certain customer needs to be queried, the correct result can be obtained only by querying from the database A and the database C.
On the basis that the database A does not complete the current synchronous task, the data in the database A and the database B form complete full data, namely target synchronous data. When the order information of a certain customer needs to be queried, the correct result can be obtained only by querying from the database A and the database B.
Further, when it is determined that the data in the database a and the database C constitute the target synchronization data, the data in the database B is emptied, and third history data generated by the remote system during a third reference period, which is a next period of time arranged in succession with the second reference period, is synchronized in real time to the database B. For example, the first reference period is 1 month and 22 days, the second reference period is 1 month and 23 days, and the third reference period is 1 month and 24 days.
When it is determined that the data in database A and database B constitute the target database, the data in database C is emptied and the third historical data generated by the remote system during the third reference is synchronized to database C in real time.
The invention empties the data in the database B or the database C, which is used for continuously storing the incremental data in the next data synchronization cycle, and ensures that the correct target synchronization data is obtained through the cyclic use of the database B and the database C. In summary, when the history databases batch synchronize the first history data within the first reference time, there are always two incremental databases as supplements, one of which is used to synchronize the first history data within the first reference period in real time, and the other of which is used to synchronize the second history data within the second reference period in real time, so that whether the history databases complete batch synchronization or not, the corresponding target incremental data can be always obtained, and the target incremental data and the data in the history databases together form complete and non-repeated full-scale data.
Still further on the basis of the above-described embodiments. Let the current time be 1 month and 23 days, synchronize all data generated by remote system 22 days in database A, synchronize incremental data generated by remote system from zero of 22 days in database B, synchronize incremental data generated by remote system 23 days in database C. Here, the mechanism of synchronizing data in the database a is different from the mechanism of synchronizing data in the database B and the database C, and the database a is all data generated the day before the synchronization at a fixed time every day, and the time required for performing the synchronization of all data is relatively long. The database B and the database C perform real-time synchronization on each piece of newly added data in the remote system, so that the synchronization speed is high and delay is almost avoided. After database a completes synchronization of all data produced by remote system 22, database C is taken as the target incremental database and the data in database B is emptied.
Assuming that the current time has reached 1 month and 24 days, at this time, database a needs to synchronize all data generated by the remote system for 23 days, and in database C, the incremental data generated by the remote system from the time when database B is idle is set, and in this case, the incremental data generated by the remote system from the time when database B is synchronized for 24 days is synchronized. At this time, if database a completes synchronization of the 23-day data, the data in database B and database a constitute complete full-size data, and database B becomes the target incremental database.
Consider another case where it is assumed that the current time has reached 1 month for 24 days, at which time database A needs to synchronize all data generated by the remote system for 23 days, and at which time database C synchronizes incremental data generated by the remote system from 23 days to zero, at which time database B is set to synchronize incremental data generated by the remote system from 24 days to zero when database B is idle. At this time, if the database a is not synchronized with the 23-day data, the data in the database C and the database a constitute complete full-amount data, and the database C becomes the target delta database.
In summary, in order to realize real-time complete synchronization of data from a remote system, the present invention respectively provides a history database and two incremental databases. Wherein the history database is used for batch synchronizing the first reference time and all data generated by the previous remote system, the first database is used for synchronizing the incremental data generated by the first reference time remote system in real time, and the second database is used for synchronizing the incremental data generated by the remote system during a second reference period, wherein a third reference period is the next time period arranged in succession with the second reference period.
Furthermore, the invention selects one of the two incremental databases as a target incremental database according to whether the historical database completes the data synchronization task, and the data in the historical database and the target incremental database form complete full data, namely target incremental data. The method for selecting the target incremental database from the two incremental databases specifically comprises the following steps: if the history database has completed the data synchronization task, selecting a second database as a target incremental database, and simultaneously clearing the data in the first database; if the history database does not complete the data synchronization task, the first database is selected as the target incremental database, and the data in the second database is emptied.
S4: and sending the target synchronous data to the local terminal.
On the basis that the target synchronous data with real-time performance and integrity is obtained, the local server sends the target synchronous data to the local terminal so that the local terminal can perform corresponding query, modification and other management operations.
With continued reference to fig. 2, in this embodiment, the big data synchronization device 10 may include or be divided into one or more program modules, where one or more program modules are stored in a storage medium and executed by one or more processors to implement the big data synchronization method of the present invention. Program modules in the present invention refer to a series of computer program instruction segments capable of performing a specific function, which are more suitable than the program itself for describing the execution of the big data synchronization device 10 in a storage medium. The following description will specifically describe functions of each program module of the present embodiment:
the state acquisition module 11 is adapted to respond to a synchronous data acquisition instruction sent by the local terminal and acquire a synchronous completion state of the local historical database; wherein the history database is used for executing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for characterizing whether the history database has completed batch synchronization of the first history data;
the increment selection module 12 is adapted to select a local first database or a local second database as a target increment database according to the synchronization completion state of the history database; wherein the first database is used for synchronizing the first historical data generated by the remote system in real time during the first reference period, and the second database is used for synchronizing the second historical data generated by the remote system in real time during a second reference period, wherein the second reference period is the next time period arranged continuously with the first reference period;
a data determining module 13 adapted to take all data in both the history database and the target delta database as target synchronization data;
and a data transmitting module 14, adapted to transmit the target synchronization data to the local terminal.
The big data synchronizing device provided by the invention, wherein the increment selecting module 13 comprises:
the first increment selection module 131 is adapted to select the second database as a target increment database if the history database has completed batch synchronization of the first history data;
the second incremental selection module 132 is adapted to select the first database as the target incremental database if the history database is not completely synchronized with the first history data.
The big data synchronizing device provided by the invention further comprises:
a first emptying module 15 adapted to empty the data in the first database after selecting the second database as a target delta database and synchronize in real time third historical data generated by a remote system during a third reference period to the first database, wherein the third reference period is a next time period arranged in succession with the second reference period;
a second emptying module 16 adapted to empty the data in the second database after selecting the first database as the target delta database and synchronize in real time third historical data generated by the remote system during a third reference to the second database.
The present embodiment also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a rack-mounted server (including an independent server or a server cluster formed by a plurality of servers) that can execute a program. The computer device 20 of the present embodiment includes at least, but is not limited to: a memory 21, a processor 22, which may be communicatively coupled to each other via a system bus, as shown in fig. 3. It should be noted that fig. 3 only shows a computer device 20 having components 21-22, but it should be understood that not all of the illustrated components are required to be implemented, and that more or fewer components may be implemented instead.
In the present embodiment, the memory 21 (i.e., readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 21 may be an internal storage unit of the computer device 20, such as a hard disk or memory of the computer device 20. In other embodiments, the memory 21 may also be an external storage device of the computer device 20, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the computer device 20. Of course, the memory 21 may also include both internal storage units of the computer device 20 and external storage devices. In this embodiment, the memory 21 is generally used to store an operating system and various types of application software installed on the computer device 20, such as program codes of the big data synchronization apparatus 10 of the first embodiment. Further, the memory 21 may be used to temporarily store various types of data that have been output or are to be output.
The processor 22 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 22 is generally used to control the overall operation of the computer device 20. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, execute the big data synchronization device 10, so as to implement the big data synchronization method of the first embodiment.
The present embodiment also provides a computer-readable storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor, performs the corresponding functions. The computer readable storage medium of the present embodiment is used for storing the big data synchronization device 10, and when executed by a processor, implements the big data synchronization method of the first embodiment.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
It will be appreciated by those of ordinary skill in the art that all or part of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, and the program may be stored in a computer readable medium, where the program when executed includes one or a combination of the steps of the method embodiment.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (5)
1. A big data synchronization method, comprising:
responding to a synchronous data acquisition instruction sent by a local terminal, and acquiring a synchronous completion state of a local historical database; wherein the history database is used for executing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for characterizing whether the history database has completed batch synchronization of the first history data;
selecting a local first database or a local second database as a target incremental database according to the synchronization completion state of the historical database; wherein the first database is used for synchronizing the first historical data generated by the remote system in real time during the first reference period, and the second database is used for synchronizing the second historical data generated by the remote system in real time during a second reference period, wherein the second reference period is the next time period arranged continuously with the first reference period;
taking all data in the history database and the target incremental database as target synchronous data;
transmitting the target synchronous data to the local terminal;
the step of selecting the first database or the second database as a target incremental database according to the synchronization completion status of the history database includes:
if the history database completes batch synchronization of the first history data, selecting the second database as a target incremental database;
if the history database is not completely synchronized with the first history data, selecting the first database as a target incremental database;
after the step of selecting the second database as the target incremental database, further comprising:
emptying data in the first database and synchronizing third historical data generated by a remote system during a third reference period to the first database in real time, wherein the third reference period is a next time period arranged in series with the second reference period;
after selecting the first database as the target incremental database, further comprising:
and (c) emptying data in the second database and synchronizing third historical data generated by a remote system during a third reference period to the second database in real time, wherein the third reference period is the next time period arranged in series with the second reference period.
2. The big data synchronization method according to claim 1, wherein the step of acquiring the synchronization completion status of the local history database comprises:
inquiring test data from the historical database, wherein the test data comprises a time field corresponding to the first reference period;
if the test data can be queried from the historical database, the historical database completes batch synchronization of the historical data;
and if the test data cannot be queried from the historical database, completing batch synchronization of the historical data by the historical database.
3. A big data synchronization device, comprising:
the state acquisition module is adapted to respond to a synchronous data acquisition instruction sent by the local terminal and acquire the synchronous completion state of the local historical database; wherein the history database is used for executing batch synchronization operation on first history data generated by a remote system during a first reference period; the synchronization completion state is used for characterizing whether the history database has completed batch synchronization of the first history data;
the increment selection module is suitable for selecting a local first database or a local second database as a target increment database according to the synchronous completion state of the historical database; wherein the first database is used for synchronizing the first historical data generated by the remote system in real time during the first reference period, and the second database is used for synchronizing the second historical data generated by the remote system in real time during a second reference period, wherein the second reference period is the next time period arranged continuously with the first reference period;
the data determining module is suitable for taking all data in the history database and the target incremental database as target synchronous data;
the data transmitting module is suitable for transmitting the target synchronous data to the local terminal;
the increment selection module includes:
the first increment selection module is suitable for selecting the second database as a target increment database if the history database completes batch synchronization of the first history data;
the second increment selection module is suitable for selecting the first database as a target increment database if the history database is not completely synchronized with the first history data;
a first flushing module adapted to flush data in the first database after selecting the second database as a target delta database and synchronize third historical data generated by a remote system during a third reference period to the first database in real time, wherein the third reference period is a next time period arranged in series with the second reference period;
and the second emptying module is suitable for emptying data in the second database after the first database is selected as a target incremental database, and synchronizing third historical data generated by a remote system during a third reference to the second database in real time.
4. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 2 when the computer program is executed by the processor.
5. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910841880.8A CN110674214B (en) | 2019-09-06 | 2019-09-06 | Big data synchronization method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910841880.8A CN110674214B (en) | 2019-09-06 | 2019-09-06 | Big data synchronization method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110674214A CN110674214A (en) | 2020-01-10 |
| CN110674214B true CN110674214B (en) | 2023-11-21 |
Family
ID=69076613
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910841880.8A Active CN110674214B (en) | 2019-09-06 | 2019-09-06 | Big data synchronization method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110674214B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113377872B (en) * | 2021-06-25 | 2024-02-27 | 北京红山信息科技研究院有限公司 | Offline synchronization method, device and equipment of online system data in big data center |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104270424A (en) * | 2014-09-17 | 2015-01-07 | 深圳创维数字技术有限公司 | Database synchronization method, server and system |
| CN104809200A (en) * | 2015-04-24 | 2015-07-29 | 联动优势科技有限公司 | Database synchronization method and device |
| CN106919664A (en) * | 2017-02-16 | 2017-07-04 | 北京诺思维信科技有限公司 | Suitable for the inquiry of historical data methods of exhibiting and system of power station Full scope simulator |
| CN107729515A (en) * | 2017-10-26 | 2018-02-23 | 咪咕文化科技有限公司 | The method, apparatus and storage medium of a kind of data syn-chronization |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7299285B2 (en) * | 2002-05-15 | 2007-11-20 | Adc Dsl Systems, Inc. | Resource sharing with database synchronization |
| US10762107B2 (en) * | 2016-11-29 | 2020-09-01 | Sap Se | Synchronization mechanism for serialized data log replay in database systems |
-
2019
- 2019-09-06 CN CN201910841880.8A patent/CN110674214B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104270424A (en) * | 2014-09-17 | 2015-01-07 | 深圳创维数字技术有限公司 | Database synchronization method, server and system |
| CN104809200A (en) * | 2015-04-24 | 2015-07-29 | 联动优势科技有限公司 | Database synchronization method and device |
| CN106919664A (en) * | 2017-02-16 | 2017-07-04 | 北京诺思维信科技有限公司 | Suitable for the inquiry of historical data methods of exhibiting and system of power station Full scope simulator |
| CN107729515A (en) * | 2017-10-26 | 2018-02-23 | 咪咕文化科技有限公司 | The method, apparatus and storage medium of a kind of data syn-chronization |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110674214A (en) | 2020-01-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113111129B (en) | Data synchronization method, device, equipment and storage medium | |
| CN109271435B (en) | Data extraction method and system supporting breakpoint continuous transmission | |
| CN111078140B (en) | Nuclear power station file uploading management method and device, terminal equipment and medium | |
| CN105302721A (en) | System and method for realizing integration testing of centralized analog server in cloud computing platform | |
| CN108446171B (en) | Electronic device, distributed system execution task allocation method and storage medium | |
| CN111400288A (en) | Data quality inspection method and system | |
| WO2019109519A1 (en) | Service rule management method, apparatus and device, and computer-readable storage medium | |
| CN112115200B (en) | Data synchronization method, device, electronic equipment and readable storage medium | |
| CN111680104B (en) | Data synchronization method, device, computer equipment and readable storage medium | |
| CN108255735B (en) | Associated environment testing method, electronic device and computer readable storage medium | |
| CN111061802B (en) | Power data management processing method, device and storage medium | |
| CN111459629A (en) | Azkaban-based project operation method and device and terminal equipment | |
| CN110674214B (en) | Big data synchronization method, device, computer equipment and storage medium | |
| CN112559525B (en) | Data checking system, method, device and server | |
| CN113360581A (en) | Data processing method, device and storage medium | |
| CN115984022B (en) | Unified account checking method and device for distributed payment system | |
| WO2019062087A1 (en) | Attendance check data testing method, terminal and device, and computer readable storage medium | |
| CN115221116A (en) | Data writing method, device and equipment and readable storage medium | |
| CN115098232A (en) | Task scheduling method, device and equipment | |
| CN113468143A (en) | Data migration method, system, computing device and storage medium | |
| CN113971074A (en) | Transaction processing method and device, electronic equipment and computer readable storage medium | |
| CN112612773A (en) | Database synchronization test method and device, computer equipment and storage medium | |
| CN112837007A (en) | Supply chain management method, device, equipment and storage medium | |
| CN111638905A (en) | Version number generation method and device, electronic equipment and readable storage medium | |
| CN116560722B (en) | Operation and maintenance flow processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |