CN115525721A - Data synchronization method, device, equipment and storage medium - Google Patents
Data synchronization method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115525721A CN115525721A CN202211252288.2A CN202211252288A CN115525721A CN 115525721 A CN115525721 A CN 115525721A CN 202211252288 A CN202211252288 A CN 202211252288A CN 115525721 A CN115525721 A CN 115525721A
- Authority
- CN
- China
- Prior art keywords
- synchronization
- data set
- data
- target data
- date
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method, a device, equipment and a storage medium, wherein the data synchronization method comprises the following steps: reading a target data set from the data configuration file, and determining the data date of the target data set; determining whether the target data set needs to be synchronized, and if so, determining the synchronization attribute information of the target data set; the synchronization attribute information includes at least a maximum synchronization period number; and synchronizing the target data set according to the data date and the maximum synchronization period. By the technical scheme, the real-time monitoring of the processing of the data sets is realized, so that the data synchronization can be carried out once the processing of the data sets is completed, the data sets are synchronized to have concurrency, and the synchronization efficiency of the data sets is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data synchronization method, apparatus, device, and storage medium.
Background
Some large application systems receive data of multiple systems for analysis and processing due to business needs. Such systems have rich data sources and are generally divided into three functional modules according to data flow: the system comprises a data mart module, a data analysis module and a data display module. In order to take account of both data Processing performance and user experience, the databases of the data mart module and the data analysis module are usually deployed in an On-Line Analytical Processing (OLAP) type database, and the database of the data presentation module is deployed in an On-Line Transaction Processing (OLTP) type database. The data mart module carries out processing such as screening, integration, mapping and the like on the data received from the upstream system; then, processing a plurality of data sets processed by the data market module in a data analysis module to generate indexes or reports, and storing the final data sets in an OLAP type database; and then displaying the data set in the data display module for the user to inquire.
When the data display module displays the data set, it is first necessary to synchronize the data set in the OLAP type database to the OLTP type database, and then display the data to the user. However, in the data synchronization method of performing data set synchronization according to the date sequence after all data sets of T days are processed, once incomplete synchronization of the data sets cannot be found in time, a large amount of data sets to be synchronized are easily accumulated, and the synchronization efficiency of the data sets is reduced.
Disclosure of Invention
The invention provides a data synchronization method, a data synchronization device, data synchronization equipment and a storage medium, which are used for improving the synchronization efficiency of data sets.
According to an aspect of the present invention, there is provided a data synchronization method, including:
reading a target data set from the data configuration file, and determining the data date of the target data set;
determining whether the target data set needs to be synchronized, and if so, determining the synchronization attribute information of the target data set; the synchronization attribute information includes at least a maximum synchronization period number;
and synchronizing the target data set according to the data date and the maximum synchronization period.
According to another aspect of the present invention, there is provided a data synchronization apparatus including:
the data set reading module is used for reading a target data set from the data configuration file and determining the data date of the target data set;
the synchronization determining module is used for determining whether the target data set needs to be synchronized, and if the target data set needs to be synchronized, determining the synchronization attribute information of the target data set; the synchronization attribute information includes at least a maximum synchronization period number;
and the data set synchronization module is used for synchronizing the target data set according to the data date and the maximum synchronization period.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data synchronization method of any of the embodiments of the invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement a data synchronization method according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the technical scheme of the embodiment of the invention, the target data set is read from the data configuration file, whether the target data set needs to be synchronized is determined, and if the target data set needs to be synchronized, the target data set is synchronized according to the data date and the maximum synchronization period number of the target data set. According to the technical scheme, the data sets are synchronized according to the data date and the maximum synchronization period, so that the real-time monitoring of data set processing is realized, the data synchronization can be performed once the data sets are processed, the data set synchronization has concurrency, and the synchronization efficiency of the data sets is improved.
It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present invention, nor are they intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1A is a flowchart of a data synchronization method according to an embodiment of the present invention;
fig. 1B is a schematic structural diagram of a data synchronization system according to an embodiment of the present invention;
fig. 2A is a flowchart of a data synchronization method according to a second embodiment of the present invention;
FIG. 2B is a flowchart illustrating data synchronization of data sets by day in a data synchronization system according to a second embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data synchronization apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device implementing the data synchronization method according to the embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It is to be understood that the terms "target" and the like in the description and claims of the present invention and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. In the technical scheme disclosed by the invention, the related data collection, storage, use, processing, transmission, provision, disclosure and the like all conform to the regulations of related laws and regulations and do not violate the common customs of the public order.
Example one
Fig. 1A is a flowchart of a data synchronization method according to an embodiment of the present invention, where the embodiment of the present invention is applicable to a situation of data set synchronization in a large application system, and the method may be executed by a data synchronization apparatus, where the data synchronization apparatus may be implemented in a form of hardware and/or software, and the data synchronization apparatus may be configured in an electronic device, and the electronic device may specifically be a data synchronization system. The data synchronization system (shown in fig. 1B) specifically includes a configuration management service 10, a data configuration monitoring service 20, a data synchronization service 30, and a data synchronization management service 40. As shown in fig. 1A, the data synchronization method includes:
s101, reading a target data set from the data configuration file, and determining the data date of the target data set.
The data configuration file may include a data set name, a data date, a synchronization date, and the like, and the batch server stored in the configuration management service 10 in the data synchronization system may be generated daily according to a service requirement. Wherein, the data date may refer to the time of the data set change; the synchronization date may refer to the time at which the data set completed the data synchronization. Data sets can be processed at any time, so that one data set can have a plurality of different data dates. When a data set is processed, the data set name, the data date, the synchronization date and the like of the data set are written into the data configuration file. Illustratively, the dataset DataA is changed at 6/1/2021, data synchronization is completed at 18/6/2021, and a piece of data is written into the data configuration file, the data set name of the data is DataA, the data date is 2021, 6/1/2021, and the synchronization date is 2021, 6/18/2021. The target data set may refer to a data set to be data synchronized.
Illustratively, in the data synchronization system, the data configuration monitoring service 20 reads the target data set from the data configuration file in the configuration management service 10 and extracts the data date of the target data set according to the data set name of the target data set. The configuration management service 10 is configured to provide configuration information for the data synchronization apparatus, and configure functions for the data synchronization system, where the configuration information may include a data configuration file, a data synchronization configuration table, a data synchronization information table, and a data table to be synchronized. The data configuration monitoring service 20 may be started on a daily basis for monitoring configuration information in the configuration management service 10, which includes time information of data sets in the data configuration file and synchronization information in the data synchronization configuration table. Meanwhile, the data configuration monitoring service 20 generates a data table to be synchronized by monitoring these configuration information.
S102, determining whether the target data set needs to be synchronized, and if so, determining the synchronization attribute information of the target data set; the synchronization attribute information includes at least a maximum number of synchronization periods.
For example, determining whether the target data set needs to be synchronized may be determined according to a pre-established data synchronization protocol, and if the target data set meets the requirements of the data synchronization protocol, determining that the target data set needs to be synchronized, otherwise, determining that the target data set does not need to be synchronized.
For example, whether the target data set needs to be synchronized may be determined according to the state of the target data set, and if the state of the target data set meets a preset data synchronization requirement state, it is determined that the target data set needs to be synchronized, and otherwise, it is determined that the target data set does not need to be synchronized.
For example, determining whether the target data set needs to be synchronized may be determined according to a type of the target data set, which may be a changed target data set and a running target data set, and if the type of the target data set is a changed target data set, determining that the target data set needs to be synchronized, and otherwise, determining that the target data set does not need to be synchronized.
The synchronization attribute information may refer to data set information that satisfies a data synchronization requirement, and the synchronization attribute information may include a data set name, a data date, and the like in addition to the maximum synchronization period. The maximum synchronization period number may refer to the maximum number of pieces of data of the synchronizable data set per synchronized batch run by the data synchronization service 30 in the data synchronization system. The maximum number of synchronization periods may be set in advance according to the service requirements. Illustratively, the maximum sync period number of the data set DataB is 5, which indicates that the data synchronization service 30 in the data synchronization system can synchronize up to 5 pieces of data of the data set DataB per time of the synchronized batch runtime. The data synchronization service 30 is configured to scan the data sets that are not synchronized or need to be resynchronized in the data configuration monitoring service 20 at regular time, perform data set synchronization, and write the synchronization execution status into the data synchronization information table for the data synchronization management service 40 to use. The data synchronization information table is stored in a target database in the configuration management service 10 in the data synchronization system.
Specifically, the data configuration monitoring service 20 in the data synchronization system determines whether the target data set needs to be synchronized, and if the target data set needs to be synchronized, acquires synchronization attribute information of the target data set. The above operation avoids invalid synchronization of the data sets.
And S103, synchronizing the target data set according to the data date and the maximum synchronization period.
Specifically, the data synchronization service 30 in the data synchronization system may sum the data date of the first time that the target data set is processed and the maximum synchronization period number to obtain the data date that the target data set can perform data synchronization, and synchronize the target data set according to the data dates, thereby implementing real-time monitoring of processing of the data set and enabling the data set that is not synchronized in multiple periods to be flexibly processed.
Illustratively, the target data set is DataC, the maximum synchronization period is 2 days, and the first time the DataC is changed is 2021 year, 7 month and 1 day, i.e. the data date is 2021 year, 7 month and 1 day, then the DataC can perform data synchronization with data dates of 2021 year, 7 month and 1 day, and 2021 year, 7 month and 2 day, and perform data synchronization with the DataC whose data date is these two days.
According to the technical scheme of the embodiment of the invention, the target data set is read from the data configuration file, whether the target data set needs to be synchronized is determined, and if the target data set needs to be synchronized, the target data set is synchronized according to the data date and the maximum synchronization period number of the target data set. According to the technical scheme, the data sets are synchronized according to the data date and the maximum synchronization period, so that the real-time monitoring of data set processing is realized, the data synchronization can be performed once the data sets are processed, the data set synchronization has concurrency, and the synchronization efficiency of the data sets is improved.
On the basis of the above embodiment, as an optional manner of the embodiment of the present invention, "synchronize a target data set according to a data date and a maximum synchronization period" is embodied as: determining the date of the data to be synchronized of the target data set according to the data date and the maximum synchronization period number; and synchronizing the target data set according to the date of the data to be synchronized.
The data date to be synchronized can be obtained by inquiring the data table to be synchronized, and the number of the data date to be synchronized is the same as the value of the maximum synchronization period number. Illustratively, the maximum number of synchronization periods is 5, then there are 5 data dates to be synchronized. The data table to be synchronized may be stored in a target database in the configuration management service 10 in the data synchronization system in fields such as a data set name, a data date, and a maximum synchronization period number, and is used to store data set information that needs to be synchronized.
Specifically, the data synchronization service 30 of the data synchronization system is operated on a daily basis, the data synchronization service 30 acquires the date of the data to be synchronized of the target data set from the data table to be synchronized according to the data date and the maximum synchronization period number of the target data set, and then the data synchronization service 30 synchronizes the data set of the target data set according to the acquired date of the data to be synchronized.
According to the technical scheme of the embodiment of the invention, the target data set is synchronized by determining the date of the data to be synchronized of the target data set, the data set range of each data set synchronization is determined, targeted data set synchronization is realized, and the synchronization efficiency of the data sets is further improved.
Example two
Fig. 2 is a flowchart of a data synchronization method according to a second embodiment of the present invention, where on the basis of the foregoing embodiment, the method may further include: determining a batch date for the target data set; meanwhile, "determining whether the target data set needs to be synchronized, and if so, determining the synchronization attribute information of the target data set" is embodied as: if the synchronization information of the target data set exists in the data synchronization information table, extracting the last synchronization date of the target data set from the data synchronization information table, determining that the target data set needs to be synchronized under the condition that the batch date is later than the last synchronization date, and extracting the maximum synchronization period number of the target data set from the data synchronization configuration table; and if the synchronization information of the target data set does not exist in the data synchronization information table, determining that the target data set needs to be synchronized, and extracting the maximum synchronization period number of the target data set from the data synchronization configuration table. As shown in fig. 2, the method includes:
s201, reading the target data set from the data configuration file, determining the data date of the target data set, and determining the batch date of the target data set.
Where the run date may refer to the time the data set was processed.
Illustratively, the data configuration monitoring service 20 of the data synchronization system reads the target data set from the data configuration file in the configuration management service 10 of the data synchronization system and determines the data date and the batch date of the target data set based on the data set name of the target data set.
S202, if the synchronization information of the target data set exists in the data synchronization information table, extracting the last synchronization date of the target data set from the data synchronization information table, determining that the target data set needs to be synchronized under the condition that the batch date is later than the last synchronization date, and extracting the maximum synchronization period number of the target data set from the data synchronization configuration table.
The data synchronization information table may include fields such as a data set name, a data date, a synchronization date, and a last synchronization date, and the data set name and the data date are used as joint primary keys for maintaining synchronization information of the data set. When a data set is synchronized, the data set name, data date, synchronization date, last synchronization date and the like of the data set are written into the data synchronization information table.
The synchronization information may refer to information required for data synchronization of the data set, such as a name of the data set, a date of the data, and a date of last synchronization. The data synchronization configuration table may include fields such as a data set name, a maximum synchronization period number, and a status, and is stored in a target database in the configuration management service 10 in the data synchronization system for maintaining the maximum synchronization number of the data set. Illustratively, for data A, up to M days of data may be synchronized per synchronized batch run. The data synchronization configuration table may be maintained by a data synchronization management service 40 in the data synchronization system. The data synchronization management service 40 modifies the information in the data synchronization configuration table according to the data set synchronization situation. Meanwhile, the data synchronization management service 40 may also be used to provide visual data synchronization information to the user.
Illustratively, the data synchronization service 30 of the data synchronization system queries the data synchronization information table for the synchronization information of the target data set, and if the query result is not null, it indicates that the synchronization information of the target data set exists in the data synchronization information table, i.e., it indicates that the target data set is not synchronized for the first time. The data synchronization service 30 further extracts the last synchronization date of the target data set from the data synchronization information table, and if the batch date is equal to or greater than the last synchronization date of the target data set, the data synchronization service 30 determines that the target data set needs to be synchronized and extracts the maximum synchronization period number of the target data set from the data synchronization configuration table.
S203, if the synchronization information of the target data set does not exist in the data synchronization information table, determining that the target data set needs to be synchronized, and extracting the maximum synchronization period number of the target data set from the data synchronization configuration table.
Illustratively, the data synchronization service 30 of the data synchronization system queries the data synchronization information table for the synchronization information of the target data set, and the query result is null, which indicates that the data synchronization information table does not contain the synchronization information of the target data set, i.e. that the target data set is synchronized for the first time. The data synchronization service 30 directly determines that the target data set needs to be synchronized and extracts the maximum number of synchronization periods for the target data set from the data synchronization configuration table.
In addition, the maximum synchronization period number can be dynamically adjusted step by step according to the size of the data set, the actual time used for synchronization and the environmental resource condition, so as to obtain the optimal maximum synchronization period number. Illustratively, the maximum number of synchronization periods for which a data set is first synchronized is set to 1 day.
And S204, synchronizing the target data set according to the data date and the maximum synchronization period.
According to the technical scheme of the embodiment of the invention, whether the target data set is synchronized for the first time is judged by detecting whether the synchronization information of the target data set exists in the data synchronization information table. If the target data set is synchronized for the first time, directly judging that the data set needs to be synchronized, and further synchronizing the target data set according to the data date and the maximum synchronization period; if the target data set is not synchronized for the first time, the target data set is judged to need to be synchronized, the last synchronization date of the target data set is further extracted from the data synchronization information table, and in the case that the batch date is larger than or equal to the last synchronization date of the target data set, the target data set is synchronized according to the data date and the maximum synchronization period number. By doing so, the data set is prevented from generating errors in the data synchronization process, the smooth execution of the data set synchronization is ensured, the accumulation of the data set is avoided, and the synchronization efficiency of the data set is improved.
On the basis of the above embodiment, as an optional way of the embodiment of the present invention, the "determining whether the target data set needs to be synchronized" is embodied as: extracting the state of a target data set from a data synchronization configuration table; the status is valid or invalid; if the state of the target data set is invalid, it is determined that the target data set does not need to be synchronized.
The state of the target data set in the data synchronization configuration table can be configured according to the service requirement, and the state comprises a valid state and an invalid state. The valid state indicates that the target data set is available for the data synchronization process, and the invalid state indicates that the target data set is not available to participate in the data synchronization process.
Specifically, the data synchronization service 30 of the data synchronization system extracts the state of the target data set from the data synchronization configuration table, determines whether the target data set needs to be synchronized according to the extracted state, determines that the target data set needs to be synchronized if the state of the target data set is valid, and determines that the target data set does not need to be synchronized if the state of the target data set is invalid.
According to the technical scheme of the embodiment of the invention, the state of the target data set is extracted from the data synchronization configuration table, and whether the target data set needs to be synchronized is determined according to the state of the target data set, so that the data sets which are not synchronized in multiple periods are flexibly processed, the synchronization of the global target data set is realized, and the synchronization efficiency of the data sets is further improved.
On the basis of the foregoing embodiment, as an optional manner of the embodiment of the present invention, the data synchronization method may further include: and after the target data sets are synchronized, writing the incidence relation among the target data sets, the synchronized data dates and the synchronized dates into a data synchronization information table.
The data date of the completed synchronization may refer to a data date corresponding to the data set of which the data synchronization is completed. The association relationship may refer to that the target data sets with the same synchronization date are stored in the data synchronization information table next to each other according to the sequence of the data dates completing synchronization.
Illustratively, the target data set is DataC, there are two data dates for completing synchronization, which are 20210701 and 20210702 respectively, and the synchronization date is 20210730, and the following data information is correspondingly written in the data synchronization information table:
| data set name | Date of data | Synchronizing dates |
| DataC | 20210701 | 20210730 |
| DataC | 20210702 | 20210730 |
According to the technical scheme of the embodiment of the invention, after the target data set is synchronized, the data synchronization information table is updated by writing the incidence relation among the target data set, the synchronized data date and the synchronized date into the data synchronization information table, so that the data synchronization system can clearly distinguish the synchronized data set from the unsynchronized data set, the repeated synchronization of the same data set is avoided, and the synchronization efficiency of the data sets is improved.
In addition, a second embodiment of the present invention further provides a flowchart for performing data synchronization on a data set by day in a data synchronization system, as shown in fig. 2B, specifically including:
and S220, generating a data configuration file according to the day.
Wherein the data profiles are stored in a batch server in the configuration management service 10.
Specifically, after the batch operation of the data set processing is finished, a data file is generated on a fixed path of the server, and meanwhile, a piece of data is inserted into the data configuration file.
Exemplarily, a batch program DATA _ SET _ a.sh of 8/20/2021 is run, and after the script is executed, a DATA a.zip DATA file is generated under/olaptoltp/path; the batch program writes a piece of data to the data configuration file Config _ DataFile _20210820. Cfg: dataA |!20210801 |)! 20210820. Wherein | |! Is a separator, the first field indicates the data set name, the second field indicates the data date, and the third field indicates the run batch date.
And S221, reading a piece of data set information from the data configuration file.
Specifically, the data configuration monitoring service 20 is started on a daily basis, and the data configuration monitoring service 20 obtains the data configuration file in the configuration management service 10 and reads a piece of data set information from the data configuration file.
S222, judging whether the data sets are synchronized for the first time, if so, synchronously executing S223 and S225, and if not, executing S228.
Specifically, the data configuration monitoring service 20 acquires the data set information and the data synchronization information table from the configuration management service 10, and then determines whether the data set is synchronized for the first time by determining whether the synchronization information of the data set exists in the data synchronization information table.
And S223, running a data synchronization management service.
The data synchronization management service 40 is configured to provide visual data synchronization information to the user, and modify information in the data synchronization configuration table according to the data set synchronization condition.
And S224, maintaining a data synchronization configuration table.
Specifically, the data synchronization management service 40 maintains a data synchronization configuration table according to the synchronization condition of the data sets.
Illustratively, the data synchronization management service 40 modifies the maximum number of synchronization periods for the data set based on the synchronization status of the data set.
And S225, writing data into the data table to be synchronized.
Specifically, if a data set is synchronized for the first time, the data set information is associated with the data synchronization configuration table, the data configuration monitoring service 20 extracts the state of the data set from the data synchronization configuration table, if the state of the data set is invalid, it is determined that the data set does not need to be synchronized, if the state of the data set is valid, the data configuration monitoring service 20 further extracts the data date and the maximum synchronization period number of the data set in the data synchronization information table, and determines the data date to be synchronized of the data set according to the data date and the maximum synchronization period number; and writing the data set name, the data date, whether to synchronize for the first time, the last synchronization date, the maximum synchronization period number and the like of the data set corresponding to the data date to be synchronized of the data set into a data table to be synchronized.
And S226, running the data synchronization service by day.
Specifically, the data synchronization service 30 is operated on a daily basis, and the data synchronization service 30 is configured to scan the data sets that are not synchronized or need to be resynchronized in the data configuration monitoring service 20 at regular time, perform data set synchronization, and write the synchronization execution status into the data synchronization information table for use by the data synchronization management service 40.
And S227, completing data synchronization and updating the data synchronization information table.
Illustratively, the data synchronization service 30 synchronizes the data set to the OLTP library and writes a piece of data to the data synchronization information table.
And S228, judging whether the data synchronization configuration table is modified or not, if so, executing S229, and if not, executing S225.
Illustratively, the state of the data set or the maximum synchronization period of the data set in the data synchronization configuration table is modified according to business requirements.
S229, the data synchronization configuration table is maintained, and then S225 is executed.
EXAMPLE III
Fig. 3 is a schematic structural diagram of a data synchronization apparatus according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes:
a data set reading module 301, configured to read a target data set from a data configuration file, and determine a data date of the target data set;
a synchronization determining module 302, configured to determine whether a target data set needs to be synchronized, and if the target data set needs to be synchronized, determine synchronization attribute information of the target data set; the synchronization attribute information includes at least a maximum synchronization period number;
a data set synchronization module 303, configured to synchronize the target data set according to the data date and the maximum synchronization period.
According to the technical scheme of the embodiment of the invention, a target data set is read from a data configuration file through a target data set reading module; determining whether the target data set needs to be synchronized through a synchronization determining module, and if so, determining the synchronization attribute information of the target data set; and synchronizing the target data set according to the data date and the maximum synchronous period number through a data set synchronizing module. According to the technical scheme, the data sets are synchronized according to the data date and the maximum synchronization period, so that the real-time monitoring of data set processing is realized, the data synchronization can be performed once the data sets are processed, the data set synchronization has concurrency, and the synchronization efficiency of the data sets is improved.
Further, the data synchronization apparatus may further include:
a batch date determination module for determining a batch date of the target data set;
the synchronization determining module 302 is specifically configured to: if the synchronization information of the target data set exists in the data synchronization information table, extracting the last synchronization date of the target data set from the data synchronization information table, determining that the target data set needs to be synchronized under the condition that the batch date is later than the last synchronization date, and extracting the maximum synchronization period number of the target data set from the data synchronization configuration table; and if the synchronization information of the target data set does not exist in the data synchronization information table, determining that the target data set needs to be synchronized, and extracting the maximum synchronization period number of the target data set from the data synchronization configuration table.
Further, the synchronization determining module 302 includes:
a state extraction unit for extracting the state of the target data set from the data synchronization configuration table; the status is valid or invalid;
and a synchronization-unneeded determination unit configured to determine that the target data set is not synchronized if the status of the target data set is invalid.
Further, the data set synchronization module 303 is specifically configured to: determining the date of the data to be synchronized of the target data set according to the data date and the maximum synchronization period number; and synchronizing the target data set according to the date of the data to be synchronized.
Further, the data synchronization apparatus may further include:
and the information table updating module is used for writing the association relation among the target data set, the synchronized data date and the synchronized date into the data synchronization information table after the target data set is synchronized.
The data synchronization device provided by the embodiment of the invention can execute the data synchronization method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the data synchronization method.
Example four
FIG. 4 shows a schematic block diagram of an electronic device 400 that may be used to implement embodiments of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 4, the electronic device 400 includes at least one processor 401, and a memory communicatively connected to the at least one processor 401, such as a Read Only Memory (ROM) 402, a Random Access Memory (RAM) 403, and the like, wherein the memory stores computer programs executable by the at least one processor, and the processor 401 may perform various suitable actions and processes according to the computer programs stored in the Read Only Memory (ROM) 402 or the computer programs loaded from a storage unit 408 into the Random Access Memory (RAM) 403. In the RAM403, various programs and data required for the operation of the electronic device 400 can also be stored. The processor 401, ROM402 and RAM403 are connected to each other by a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
A number of components in the electronic device 400 are connected to the I/O interface 405, including: an input unit 406 such as a keyboard, a mouse, or the like; an output unit 407 such as various types of displays, speakers, and the like; a storage unit 408 such as a magnetic disk, optical disk, or the like; and a communication unit 409 such as a network card, modem, wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
In some embodiments, the data synchronization method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 400 via the ROM402 and/or the communication unit 409. When the computer program is loaded into RAM403 and executed by processor 401, one or more steps of the data synchronization method described above may be performed. Alternatively, in other embodiments, the processor 401 may be configured to perform the data synchronization method by any other suitable means (e.g., by way of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the Internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired result of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for synchronizing data, the method comprising:
reading a target data set from a data configuration file, and determining the data date of the target data set;
determining whether the target data set needs to be synchronized, and if so, determining the synchronization attribute information of the target data set; the synchronization attribute information at least comprises a maximum synchronization period number;
and synchronizing the target data set according to the data date and the maximum synchronization period number.
2. The method of claim 1, further comprising: determining a batch date for the target dataset;
the determining whether the target data set needs to be synchronized, and if so, determining synchronization attribute information of the target data set includes:
if the synchronization information of the target data set exists in a data synchronization information table, extracting the last synchronization date of the target data set from the data synchronization information table, determining that the target data set needs to be synchronized under the condition that the batch date is later than the last synchronization date, and extracting the maximum synchronization period number of the target data set from a data synchronization configuration table;
and if the synchronization information of the target data set does not exist in the data synchronization information table, determining that the target data set needs to be synchronized, and extracting the maximum synchronization period number of the target data set from the data synchronization configuration table.
3. The method of claim 1, wherein the determining whether the target data set requires synchronization comprises:
extracting the state of the target data set from a data synchronization configuration table; the status is valid or invalid;
determining that the target data set does not need to be synchronized if the status of the target data set is invalid.
4. The method of claim 1, wherein synchronizing the target data set according to the data date and a maximum number of synchronization periods comprises:
determining the date of the data to be synchronized of the target data set according to the data date and the maximum synchronization period number;
and synchronizing the target data set according to the date of the data to be synchronized.
5. The method of claim 1, wherein after synchronizing the target data set, further comprising:
and writing the incidence relation among the target data set, the data date completing synchronization and the synchronization date into a data synchronization information table.
6. A data synchronization apparatus, comprising:
the data set reading module is used for reading a target data set from a data configuration file and determining the data date of the target data set;
a synchronization determining module, configured to determine whether the target data set needs to be synchronized, and if the target data set needs to be synchronized, determine synchronization attribute information of the target data set; the synchronization attribute information at least comprises a maximum synchronization period number;
and the data set synchronization module is used for synchronizing the target data set according to the data date and the maximum synchronization period number.
7. The apparatus of claim 6, further comprising: determining a batch date for the target dataset;
the synchronization determining module is specifically configured to:
if the synchronization information of the target data set exists in a data synchronization information table, extracting the last synchronization date of the target data set from the data synchronization information table, determining that the target data set needs to be synchronized under the condition that the batch date is later than the last synchronization date, and extracting the maximum synchronization period number of the target data set from a data synchronization configuration table;
and if the synchronization information of the target data set does not exist in the data synchronization information table, determining that the target data set needs to be synchronized, and extracting the maximum synchronization period number of the target data set from the data synchronization configuration table.
8. The apparatus of claim 6, wherein the synchronization determination module comprises:
a state extraction unit, configured to extract a state of the target data set from a data synchronization configuration table; the status is valid or invalid;
a synchronization unneeded determination unit configured to determine that the target data set is not synchronized if the status of the target data set is invalid.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data synchronization method of any one of claims 1-5.
10. A computer-readable storage medium having stored thereon computer instructions for causing a processor to perform the data synchronization method of any one of claims 1-5 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211252288.2A CN115525721A (en) | 2022-10-13 | 2022-10-13 | Data synchronization method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211252288.2A CN115525721A (en) | 2022-10-13 | 2022-10-13 | Data synchronization method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115525721A true CN115525721A (en) | 2022-12-27 |
Family
ID=84701835
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211252288.2A Pending CN115525721A (en) | 2022-10-13 | 2022-10-13 | Data synchronization method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115525721A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115718787A (en) * | 2023-01-09 | 2023-02-28 | 百融至信(北京)科技有限公司 | Data table data synchronization method, query method, electronic equipment and storage medium |
-
2022
- 2022-10-13 CN CN202211252288.2A patent/CN115525721A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115718787A (en) * | 2023-01-09 | 2023-02-28 | 百融至信(北京)科技有限公司 | Data table data synchronization method, query method, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110647579A (en) | Data synchronization method and device, computer equipment and readable medium | |
| CN111651519B (en) | Data synchronization method, data synchronization device, electronic equipment and storage medium | |
| CN113193947B (en) | Method, apparatus, medium, and program product for implementing distributed global ordering | |
| US11442930B2 (en) | Method, apparatus, device and storage medium for data aggregation | |
| CN114925143A (en) | Method, device, equipment, medium and product for describing logical model blood relationship | |
| CN115525721A (en) | Data synchronization method, device, equipment and storage medium | |
| CN115640300A (en) | Big data management method, system, electronic equipment and storage medium | |
| CN115237426A (en) | Method, device and equipment for determining database difference and storage medium | |
| CN116414814A (en) | Data checking method, device, equipment, storage medium and program product | |
| CN115982273A (en) | Data synchronization method, system, electronic equipment and storage medium | |
| CN115408111A (en) | Database script control method, system, device, storage medium and program product | |
| CN112783980B (en) | Data synchronous processing method, device, electronic equipment and computer readable medium | |
| CN115544010A (en) | Mapping relation determining method and device, electronic equipment and storage medium | |
| CN115454971A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN114968950A (en) | Task processing method and device, electronic equipment and medium | |
| CN114443772A (en) | Distributed data processing method, device, equipment and medium | |
| CN114328739A (en) | Data synchronization method, data reading method, data synchronization device, data reading device, electronic equipment, storage medium and product | |
| CN113377872A (en) | Offline synchronization method, device and equipment of online system data in big data center | |
| CN114818635A (en) | Data report generation method and device, electronic equipment and storage medium | |
| CN113569144B (en) | Method, device, equipment, storage medium and program product for searching promotion content | |
| CN114416881A (en) | Real-time synchronization method, device, equipment and medium for multi-source data | |
| CN115687529A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN114661918A (en) | Knowledge graph construction method and device, storage medium and electronic equipment | |
| CN118132536A (en) | Data migration method, device, equipment and storage medium | |
| CN114925246A (en) | Graph data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |