CN108052681A - The synchronous method and system of structural data between a kind of relevant database - Google Patents
The synchronous method and system of structural data between a kind of relevant database Download PDFInfo
- Publication number
- CN108052681A CN108052681A CN201810030156.2A CN201810030156A CN108052681A CN 108052681 A CN108052681 A CN 108052681A CN 201810030156 A CN201810030156 A CN 201810030156A CN 108052681 A CN108052681 A CN 108052681A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- export
- entry
- json
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
The invention discloses a kind of synchronous method of structural data between relevant database, the data that the method is used to implement between source database and target database are synchronous；The described method includes：Step 1) is according to practical business application demand, to needing derived structuring initial data extraction key-value pair information in the log information of source database and source database, the intermediate data for meeting call format is rewritten into reference to data action type and data entry tag format, intermediate export data are saved as with json forms；Step 2) imports strategy according to the data of target database, export data among json are mapped as target data from basic data with reference to format conversion operation between data cleaning operation and database, and the data to be imported for meeting associated databases import format are converted into, it is then introduced into target database；Step 3) carries out resolving inversely operation to the data combining target type of database to be imported that step 2) generates, and generation is used for the Backup and Restore data of versions of data rollback.
Description
Technical field
The present invention relates to a kind of the same of the structural data between synchronous processing field of data more particularly to relevant database
One step process and system.
Background technology
In data processing field, it usually needs carry out the flows such as extraction, Mapping and Converting, the synchronization of data.Wherein original number
According to archive storage be generally stored inside the traditional databases such as MySQL, MariaDB, SQLServer, and support large-scale Search Requirement,
Further data processing stores data in the professional services such as elasticsearch, solr in the pass of data retrieval engine again
It is in type database, therefore, data needs are mapped between a variety of relevant databases, is synchronous.Further it is provided that large data
Enterprise, company and data analysis mechanism of application etc., since the authority managing and controlling of data resource is commonly provided with the safety of intranet and extranet
Isolation is set, and the requirement of offline unidirectional delivery is just proposed to the synchronization of data.
The data that traditional data synchronization scenario can well solve same type of database are synchronous, however avoid
Data mapping synchronisation requirement between disparate databases；Meanwhile progress two-way communication can safeguard number well under consolidated network
According to synchronous uniformity, but be difficult timely and effectively solve one-way communication limitation under due to the generation number of versions of data conflict
According to Backup and Restore problem.
The content of the invention
It is an object of the invention to be directed to the synchronization of the structural data between different relevant databases and exist single
To the data processing needs such as synchronous of the off-line structurals data such as communication, structural data between a kind of relevant database is proposed
Synchronous method and system, suitable for solving the data synchronization problems between database, between can completing database, data are broadly
Synchronization, i.e., the not quite identical mapping after the crash consistency including data and interpolation data cleaning operation is synchronous；Especially
The data that it is suitble to complete under off-line case are synchronous, the operation of data resolving inversely added, for versions of data rollback；Ensure number
According to reliability and stability.
To achieve these goals, the present invention proposes a kind of synchronous method of structural data between relevant database,
The data that the method is used to implement between source database and target database are synchronous；Including but not limited to same type database
Between data crash consistency between distinct type data-base it is synchronous and to combine the data transformation of data cleaning operation consistent
Property it is synchronous；The described method includes：
Step 1) is derived to being needed in the log information of source database and source database according to practical business application demand
Structuring initial data extracts key-value pair information, is rewritten into reference to data action type and data entry tag format and meets form
It is required that intermediate data, intermediate export data are saved as with json forms；
Step 2) imports strategy according to the data of target database, with reference to format conversion between data cleaning operation and database
Export data among json are mapped as target data by operation from basic data, and are converted into and are met associated databases import format
Data to be imported, be then introduced into target database；
Step 3) carries out resolving inversely operation, generation to the data combining target type of database to be imported that step 2) generates
For the Backup and Restore data of versions of data rollback.
As a kind of improvement of the above method, the step 1) specifically includes：
Step 1-1) structuring initial data derived from the needs in source database is divided into：Data table information, full word
Segment data and partial data；
Step 1-2) data table information is exported：The structural information that derived tables of data is needed in source database is exported,
Extraction key-value pair information is simultaneously converted into the preservation of json forms；The structural information of the tables of data includes：Database name, table name, volume
Code and the field name of all fields, type, length；
Step 1-3) full word segment data is exported：It is intercepted out according to the query statement that set interval generates in source database
Entry is chosen, while parses the item for needing to carry out data deletion for " deletion " operational orientation from source database log information
The structuring initial data for choosing entry is converted into key-value pair form by mesh, is generated according to specified data entry tag format
Unique identifier, as the Data Identification code of each data entry, and flag data action type, json forms are saved as, are generated
Export data among json；Wherein, if data entry tag format is not specified, it is set to sky；The number chosen from source database
It is defaulted as " increasing " according to the data manipulation type of entry, the data behaviour for the data entry chosen from the log information of source database
Make type as " deletion ", corresponding data manipulation code is set according to corresponding type of database；
Step 1-4) partial data is exported：List of fields, the query range of setting specified according to user generate inquiry
Sentence, matched data obtains choosing entry in source database, while is operated from source database log information for " deletion "
Orientation parses the entry for needing to carry out data deletion, and the structuring initial data for choosing entry is converted into key-value pair form,
Unique identifier is generated according to specified data entry tag format simultaneously, as the Data Identification code of each data entry, and
Flag data action type is saved as json forms, data is exported among generation json；Wherein, if data entry tag format not
It specifies, is then set to sky；The data manipulation type for the data entry chosen from source database is defaulted as " changing ", from source database
Log information in the data manipulation type of data entry chosen be for " deletions ", difference is set according to correspondence database type
Data manipulation code.
As a kind of improvement of the above method, the step 2) specifically includes：
Step 2-1) by export data in data Type division among json be：Tables of data creates, data import；
Step 2-2) tables of data establishment：The structural information of tables of data in export data among json is reduced into key assignments
Right, the form according to needed for target database type is filled with the new table of target database establishment creates new table to target database；
Step 2-3) data importing：Combining target type of database and user's specified data Format adjusting, data cleansing
Strategy generates data processing policy；
Step 2-4) according to step 2-3) in generation data processing policy, with reference to data command code and be equipped with data mark
Know code to carry out data format adjustment and data cleansing and generate final data import statement, import target database.
As a kind of improvement of the above method, the step 3) specifically includes：
Step 3-1) to step 2) generation data in data type to be imported be divided into data table information resolving inversely and
The resolving inversely of content-data；
Step 3-2) data table information resolving inversely：Read step 2-2) generation for creating the new table of target database
Data to be imported, be decoded as respective table cancel statement, generation for follow-up data version rollback Backup and Restore data；
Step 3-3) content-data resolving inversely：Read step 2-4) generation waits to lead for importing target database
Enter data, it is extensive for the backup of follow-up data version rollback that combining target type of database carries out corresponding resolving inversely operation generation
Complex data.
As a kind of improvement of the above method, the resolving inversely operation includes：Data entry to be imported is read, to wherein
Data manipulation code carries out reverse Mapping：" increase " operation is replaced with into " deletion " operation, correspondence is replaced with to " modification " operation tax
" modification " operation of former data, " increase " that corresponding data is replaced with to " deletion " operation operate, and generate and use with reference to entry contents
In the Backup and Restore data of follow-up data version rollback.
The present invention also provides a kind of synchronization system of structural data between relevant database, the synchronization system bag
It includes：Data synchronization engine 10, data processing module 20, message scheduling module 30, data backup storage 40 and log management storehouse 50；
The data synchronization engine 10, for being responsible for the interbehavior between user and system, including task customization, user
Rights management, the upload of data are downloaded and the expansion of other external interfaces linking；
The data processing module 20, for receiving the data processing task order of the transmission of message scheduling module 30, according to
Task order reads the data of required by task from log management storehouse 50, and data from source database are exported, and import target
In database；And resolving inversely operation is carried out to data, for the rollback of versions of data；
The message scheduling module 30, for obtaining the task requests of user configuration from data synchronization engine 10 and matching somebody with somebody
The normalization data synchronous task request put, and transfer data processing task order to data processing module 20；
The data backup storage 40 adjusts sum number for storing the data format that user uploads in data synchronization engine 10
According to the data after cleaning, the inverse of export data packet, the importing data packet of upload and generation that data processing module 20 generates is stored
To recovery data packets, running log file is determined in the journal file and each database that are generated in storage system operational process
Phase extracts the log information file for all delete operations that parsing obtains；
The log management storehouse 50, for managing generated all information exchange daily records in system operation, including
The data of task customized information, user's usage record, upload in data synchronization engine 10 and the registration of code store path, outside
The data input of interface, it is the situation record of the data exporting in data processing module 20, the data packet imported and exported, reverse
The tasks carrying information note of related information record and message scheduling module 30 between the data packet and assignment database of recovery
Record.
As a kind of improvement of above system, the data synchronization engine 10 includes：
Task customization units configure for the timed task to normalization, and disposable temporary duty is configured,
The data manipulation code that configuration is included in the merging of multiple subtasks, export process is specified, Data Identification code form configures, source data
Attribute configuration, the data format adjustment in importing process and the upload of data cleaning processing unit, the attribute of target data match somebody with somebody
It puts etc. and is configured with the calling of external interface；
The rights management unit of user is used to implement the highest priority assignation of the data synchronization of administrator, maintenance personnel
It performs permission, the provisional configuration of data application personnel and calls permission；
Interactive interface is factually now uploaded for logarithm and downloaded；With the modification and upload of data cleaning processing unit；Task is determined
The interaction such as system and attribute configuration；
External interface connects for the application range and expanding for system intelligent service of increase of growth data synchronization system
Mouth management, if the increasing of data import unit and data lead-out unit changes, CD ferry-boat system docking of data flow control etc..
As a kind of improvement of above system, the data processing module 20 includes：Data export operating unit A, data
Import operation unit B and resolving inversely recovery unit C；
The data export operating unit A, are used to implement initial data and export among json is exported to from source database
Data, at the same read data backup storage 40 on " deletions " operate source database log information, parse need into
The export data for the entry generation flag bit delete operation that row data are deleted, are merged among json in export data；It will be original
Data in data type is divided into the export of tables of data, the export of full word segment data and partial data export；Task is exported according to data
Exporting policy derived initial data will be wanted to carry out the json digitizations that table creates information, full word segment data and partial data
Initial data combination data action type and data entry tag format data normalization, so as to generate export among json
Data；
The data import operation unit B is used to implement export data among json importing target database；By json
Centre imports data in data type and is divided into the establishment of tables of data and the importing of data to be imported；By data table information to be imported
The requirement that according to target the new table of database creates carries out data format, and content-data to be imported is pressed data cleansing and number of targets
Requirement is imported according to storehouse and completes data mapping, and target database is imported so as to generate final import statement；
The resolving inversely recovery unit C, for data to be imported to be carried out with resolving inversely operation generation for data version
The Backup and Restore data of this rollback.
Advantage of the invention is that：
The method and system of the present invention causes data synchronous service of the user between different relevant databases to implement more
It is convenient to add, and the data for solving the problems, such as the lower reply versions of data conflict of one-way communication limitation using data recovery policy are recovered, can
A variety of different data synchronization scenarios are successfully managed, it can also be that more intelligentized data synchronous system is established to be expanded by interface
Basis.
Description of the drawings
Fig. 1 is the schematic diagram of the method for the present invention；
Fig. 2 is the schematic diagram of the system of the present invention.
Specific embodiment
The present invention will be described in detail in the following with reference to the drawings and specific embodiments.
As shown in Figure 1, data are synchronously broadly divided into two steps, first, data export, data are led from source database
Go out：According to practical business application demand, structural data extraction key-value pair information derived from the needs in source database is rewritten
Into the intermediate data for meeting call format, json texts are saved as, to make follow-up use or backup；Second is that data import, by data
Target database is imported, and generates the perform script of data recovery：According to the signal intelligence and target data of target database
Storehouse data import strategy, and the data of json intermediate data texts are imported database, are carried out at the same time the resolving inversely of text data
Backup and Restore data of the generation available for versions of data rollback.
(1) data export
Data synchronously move to all data consistencies in source database in target database, farthest
Keep the uniformity of the master data of two databases.Derived data are broadly divided into the table structure export of data, full word hop count
According to export, partial data export three classes, wherein data export is usually directed to " increase ", " deletion ", " modification " three kinds of operations, in number
According in export process, it is necessary to preserve the operation information of three kinds of different operatings.
The table structural information export section of data is the establishment code information that derived tables of data will be needed in source database
Export, extraction key-value pair information are converted into the preservation of json forms.The structural information of tables of data mainly includes：Database name, table name,
Field name, type, length of coding and all fields etc..In establishment information, elasticsearch such as the table in sql
The schema of core in the mapping information and solr of type.
Since the data delete operation to database is without data synchronous system, but for the deletion of database data
Operation extracts in database journal since feature is substantially easier differentiation, so herein can be according to database journal
The daily record that orientation parsing gather data is deleted, holds so as to generate the data delete operation information in source database on specific table
This is traveled far and wide, while the data manipulation code of different mark delete operations is set according to correspondence database type.The information of this part
Specific delete operation daily record detection program can be created on database server, identifies and synthesizes for specific export process
Into the export text of generation.
The export section of full word segment data, generating query statement according to set interval and intercept out needs to export in source database
Part, the structural data for choosing entry is converted into key-value pair form and is saved as the export of json forms, while according to specifying
Data entry tag format is equipped with a data identification code for each data entry, if not specified, is set to sky；Acquiescence behaviour herein
Make information as " increasings ", according to the corresponding data manipulation code of corresponding type of database setting, as in sql insert,
Add in index, solr in elasticsearch etc. is to distinguish, if special requirement is according to export data update number
It could be provided as " changing " according to table, data manipulation code is set under.
The export section of partial data, derived list of fields and set interval generate corresponding query statement as needed
Intercepting out needs derived part in source database, the structural data for choosing entry is converted into key-value pair form is saved as json
Form exports, while is equipped with a data identification code according to specified data entry tag format for each data entry, if not
It specifies, is then set to sky；Default action information is " modification ", different data manipulation codes is set according to correspondence database type, such as
Add_field of update, solr of update_set, elasticsearch of sql etc. is to distinguish, if according to leading
Go out data newdata table to could be provided as " increasing ", the setting of data manipulation code is same as above.
(2) data import
Data import the data obtained primarily directed to data export link and are parsed, and are entered into target database
In.It is divided into according to data type to be imported and the tables of data of target database is created, based on the synchronization of table data based on list structured data
Two class of targeted database data.
The lead-in portion of new table is created based on list structured data, is that the json structural informations imported in data are parsed into bonding
Value pair, the form according to needed for target database type is filled with the new table of target database establishment, and respective table is called to create mould
Block is created, while generates the Backup and Restore data for follow-up data version rollback for deleting tables of data.
It is by the json data 1 of importing based on the lead-in portion of table data synchronization target database) according to target database
Type of database call corresponding data import unit, 2) according to data such as specified data Format adjusting, data cleansings at
Reason cell processing data entry to be imported, 3) it is mapped together according to data manipulation code and the Data Identification code being equipped with progress data
Step；The data of corresponding data convert recovery are generated according to data manipulation code and data identification code resolving inversely simultaneously, such as to " increasing
Add " it operates and implements " deletion " operation, " modification " operation is implemented to operate based on " modification " that corresponds to former data, " deletion " is operated
Implement " increase " operation of corresponding data, so as to generate the perform script of the data recovery for data convert.
As shown in Fig. 2, data synchronous system mainly includes data synchronization engine 10, data processing module 20, scheduling message
Module 30, data backup storage 40, log management storehouse 50.
(1) data synchronization engine 10
Data synchronization engine 10 be mainly responsible for user it is synchronous with data between all interbehaviors, including task customization,
The rights management of user, the upload of data are downloaded and the expansion of other external interfaces is connected such as：The data of one-way communication are synchronous
Involved in CD ferry-boat application etc..
Wherein task customization is divided into two big types：The configuring of the timed task of normalization, the configuration of disposable temporary duty,
The data manipulation code that configuration is included in the merging of multiple subtasks, export process is specified, Data Identification code form configures, source data
Attribute configuration, the data format adjustment in importing process and the upload of data cleaning processing unit, the attribute of target data match somebody with somebody
It puts etc. and is configured with the calling of external interface；
The rights management of user is related to the highest permission of the data synchronization of administrator, execution permission, the data of maintenance personnel
Permission etc. is called using the provisional configuration of personnel；
It is most basic system interaction interface that the upload of data, which is downloaded,；With the modification and upload of data cleaning processing unit；
Task customizes interactive with attribute configuration etc.；
The linking of other external interfaces is the application range of growth data synchronization system and increases system intelligent service
Interface management can be expanded, if the increasing of data import unit and data lead-out unit changes, the CD ferry-boat system of data flow control
Docking etc..
(2) data processing module 20
The data processing module 20 includes：Data export operating unit A, data import operation unit B and resolving inversely
Recovery unit C；
The data export operating unit A, are used to implement initial data and export among json is exported to from source database
Data, while the source database log information on delete operation periodically extracted read in data backup storage 40 parses
The entry for carrying out data deletion is needed to generate the export data of flag bit delete operation, is merged among json in export data；
Export, the export of full word segment data and partial data including initial data to be divided into data table information by data type export.Root
According to the exporting policy of data export task derived initial data will be wanted to carry out the json digitizations that table creates information, full word section
Data and the initial data combination data action type of partial data and the data normalization of data entry tag format, so as to raw
Export data among into json；
The data import operation unit B is used to implement export data among json importing target database；Including inciting somebody to action
Data in data type is imported among json and is divided into the establishment of tables of data, the importing of data to be imported.Task is imported according to data
Importing strategy, export data among json are divided into：According to target the new table of database creates requirement to standard scale information to be imported
Data format, standard content data in data to be imported cleaning and target database import requirement data mapping, from
And it generates final import statement and imports target database；
The resolving inversely recovery unit C is subsequently used for data version for data to be imported to be carried out with resolving inversely generation
The Backup and Restore data of this rollback.Including to carrying out corresponding contrary operation class under data combination disparate databases type to be imported
Type maps and the Backup and Restore entry data of corresponding data entry is generated in individual event communication network, and data export and importing are the same as two-way
Offline synchronization in communication network.
(3) message scheduling module 30
Task requests that the user that message scheduling module 30 is responsible for obtaining in gather data synchronization engine 10 newly configures and
The normalization data synchronous task request of configuration, and transfer data processing task order to data processing module 20.Scheduling message
Module can avoid message congestion well, so as to mitigate system pressure and reduce the mistake or something lost of data processing task execution
Leakage.
(4) data backup storage 40
Data backup storage 40 mainly stores the data format adjustment that user uploads in data synchronization engine 10 and data are clear
Processing unit is washed, the export data packet generated in data processing task, the importing data packet, the backward recovery data of generation that upload
It wraps, periodically extracting parsing to running log file in the journal file and each database that are generated in system operation obtains
All delete operations log information file.
(5) log management storehouse 50
Generated all information exchange daily records in record management system operational process are responsible in log management storehouse 50, including number
According to task customized information, user's usage record, the data uploaded and the code store path in synchronization engine 10 are registered, outside connects
Mouthful data input, it is the situation record of the data exporting in data processing module 20, the data packet imported and exported, reverse extensive
Related information record between multiple data packet and assignment database etc., the tasks carrying information record of message scheduling module 30.
It should be noted last that the above embodiments are merely illustrative of the technical solutions of the present invention and it is unrestricted.Although ginseng
The present invention is described in detail according to embodiment, it will be understood by those of ordinary skill in the art that, to the technical side of the present invention
Case is modified or replaced equivalently, and without departure from the spirit and scope of technical solution of the present invention, should all be covered in the present invention
Right among.
Claims (8)
1. the synchronous method of structural data, the method are used to implement source database and number of targets between a kind of relevant database
It is synchronous according to the data between storehouse；Data complete one including but not limited between same type database between distinct type data-base
Cause property is synchronous and combines the data transformation consistency synchronization of data cleaning operation；The described method includes：
Step 1) is according to practical business application demand, to needing derived structure in the log information of source database and source database
Change initial data extraction key-value pair information, be rewritten into reference to data action type and data entry tag format and meet call format
Intermediate data, intermediate export data are saved as with json forms；
Step 2) imports strategy according to the data of target database, with reference to format conversion operation between data cleaning operation and database
Export data among json are mapped as target data from basic data, and is converted into and meets treating for associated databases import format
Data are imported, are then introduced into target database；
Step 3) carries out resolving inversely operation to the data combining target type of database to be imported that step 2) generates, and generation is used for
The Backup and Restore data of versions of data rollback.
2. the synchronous method of structural data between relevant database according to claim 1, which is characterized in that the step
It is rapid 1) to specifically include：
Step 1-1) structuring initial data derived from the needs in source database is divided into：Data table information, full word hop count
According to and partial data；
Step 1-2) data table information is exported：The structural information that derived tables of data is needed in source database is exported, extraction
Key-value pair information is simultaneously converted into the preservation of json forms；The structural information of the tables of data includes：Database name, table name, coding with
And the field names of all fields, type, length；
Step 1-3) full word segment data is exported：Choosing in source database is intercepted out according to the query statement that set interval generates
Entry, while the entry for needing to carry out data deletion is parsed for " deletion " operational orientation from source database log information,
The structuring initial data for choosing entry is converted into key-value pair form, is generated according to specified data entry tag format unique
Identifier, as the Data Identification code of each data entry, and flag data action type, json forms are saved as, generate json
Centre export data；Wherein, if data entry tag format is not specified, it is set to sky；The data strip chosen from source database
Purpose data manipulation type is defaulted as " increasing ", the data manipulation class for the data entry chosen from the log information of source database
Type is " deletion ", and corresponding data manipulation code is set according to corresponding type of database；
Step 1-4) partial data is exported：List of fields, the query range of setting specified according to user generate query statement,
Matched data obtains choosing entry in source database, while " deletion " operational orientation solution is directed to from source database log information
The entry for needing to carry out data deletion is precipitated, the structuring initial data for choosing entry is converted into key-value pair form, while root
Unique identifier is generated according to specified data entry tag format, as the Data Identification code of each data entry, and reference numerals
According to action type, json forms are saved as, data are exported among generation json；Wherein, if data entry tag format is not specified,
It is set to sky；The data manipulation type for the data entry chosen from source database is defaulted as " changing ", from the daily record of source database
The data manipulation type for the data entry chosen in information is " deletion ", according to correspondence database type different data is set to grasp
Make code.
3. the synchronous method of structural data between relevant database according to claim 1, which is characterized in that the step
It is rapid 2) to specifically include：
Step 2-1) by export data in data Type division among json be：Tables of data creates, data import；
Step 2-2) tables of data establishment：The structural information of tables of data in export data among json is reduced into key-value pair, root
The form needed for the new table of target database establishment is filled with according to target database type, new table is created to target database；
Step 2-3) data importing：Combining target type of database and user's specified data Format adjusting, data cleansing strategy,
Generate data processing policy；
Step 2-4) according to step 2-3) in generation data processing policy, with reference to data command code and be equipped with Data Identification code
It carries out data format adjustment and data cleansing and generates final data import statement, import target database.
4. the synchronous method of structural data between relevant database according to claim 3, which is characterized in that the step
It is rapid 3) to specifically include：
Step 3-1) it is divided into the resolving inversely and content of data table information to the data in data type to be imported of step 2) generation
The resolving inversely of data；
Step 3-2) data table information resolving inversely：Read step 2-2) generation for creating treating for the new table of target database
Data are imported, are decoded as respective table cancel statement, generation is used for the Backup and Restore data of follow-up data version rollback；
Step 3-3) content-data resolving inversely：Read step 2-4) what is generated is used to import the number to be imported of target database
According to combining target type of database carries out Backup and Restore number of the corresponding resolving inversely operation generation for follow-up data version rollback
According to.
5. the synchronous method of structural data between relevant database according to claim 4, which is characterized in that described inverse
Include to parsing operation：Data entry to be imported is read, reverse Mapping is carried out to wherein data manipulation code：" increase " operation is replaced
It is changed to " deletion " to operate, assign " modification " operation " modification " operation for replacing with corresponding former data, " deletion " operation is replaced with
" increase " operation of corresponding data, with reference to entry contents generation for the Backup and Restore data of follow-up data version rollback.
6. the synchronization system of structural data between a kind of relevant database, which is characterized in that the synchronization system includes：Data
Synchronization engine (10), data processing module (20), message scheduling module (30), data backup storage (40) and log management storehouse
(50)；
The data synchronization engine (10), for being responsible for the interbehavior between user and system, the power including task customization, user
Limit management, the upload of data are downloaded and the expansion of other external interfaces linking；
The data processing module (20), for receiving the data processing task order of message scheduling module (30) transmission, according to
Task order reads the data of required by task from log management storehouse (50), and data from source database are exported, and import mesh
It marks in database；And resolving inversely operation is carried out to data, for the rollback of versions of data；
The message scheduling module (30), for obtaining the task requests of user configuration from data synchronization engine (10) and matching somebody with somebody
The normalization data synchronous task request put, and transfer data processing task order to data processing module (20)；
The data backup is stored in a warehouse (40), and sum number is adjusted for storing the data format that user uploads in data synchronization engine (10)
According to the data after cleaning, export data packet, the importing data packet of upload and generation that data processing module (20) generates are stored
Backward recovery data packet, to running log file in the journal file and each database that generate in storage system operational process
Periodically extract the log information file for all delete operations that parsing obtains；
The log management storehouse (50), for managing generated all information exchange daily records in system operation, including number
It is registered according to task customized information, user's usage record, the data uploaded and the code store path in synchronization engine (10), is external
The data input of interface, it is the situation record of the data exporting in data processing module (20), the data packet imported and exported, inverse
The tasks carrying letter of related information record and message scheduling module (30) between the data packet of recovery and assignment database
Breath record.
7. the synchronization system of structural data between relevant database according to claim 6, which is characterized in that the number
Include according to synchronization engine (10)：
Task customization units configure for the timed task to normalization, and disposable temporary duty is configured, configuration
Data manipulation code in merging, export process including multiple subtasks is specified, Data Identification code form configures, the category of source data
Property configuration, the data format adjustment in importing process and the upload of data cleaning processing unit, the attribute configuration etc. of target data,
And it is configured with the calling of external interface；
The rights management unit of user is used to implement the execution of the highest priority assignation, maintenance personnel of the data synchronization of administrator
Permission, the provisional configuration of data application personnel and calling permission；
Interactive interface is factually now uploaded for logarithm and downloaded；With the modification and upload of data cleaning processing unit；Task customize and
The interaction such as attribute configuration；
External interface, for growth data synchronization system application range and increase system intelligent service and expand mouthpiece
Reason, if the increasing of data import unit and data lead-out unit changes, CD ferry-boat system docking of data flow control etc..
8. according to the synchronization system of structural data between the relevant database described in claim 6, which is characterized in that the data
Processing module (20) includes：Data export operating unit A, data import operation unit B and resolving inversely recovery unit C；
The data export operating unit A, are used to implement initial data and export data among json are exported to from source database,
The source database log information on " deletion " operation in data backup storage (40) is read simultaneously, and parsing needs into line number
According to the export data of the entry generation flag bit delete operation of deletion, it is merged among json in export data；By initial data
It is divided into the export, the export of full word segment data and partial data export of tables of data by data type；Leading for task is exported according to data
Go out the original of the json digitizations that strategy will want derived initial data carry out table to create information, full word segment data and partial data
The data normalization of beginning data combination data action type and data entry tag format, so as to generate export number among json
According to；
The data import operation unit B is used to implement export data among json importing target database；Among json
It imports data in data type and is divided into the establishment of tables of data and the importing of data to be imported；Data table information to be imported is pressed into mesh
It marks the requirement that the new table of database creates and carries out data format, content-data to be imported is pressed into data cleansing and target database
It imports requirement and completes data mapping, target database is imported so as to generate final import statement；
The resolving inversely recovery unit C returns for data to be imported to be carried out with resolving inversely operation generation for versions of data
The Backup and Restore data of rolling.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810030156.2A CN108052681B (en) | 2018-01-12 | 2018-01-12 | Method and system for synchronizing structured data between relational databases |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810030156.2A CN108052681B (en) | 2018-01-12 | 2018-01-12 | Method and system for synchronizing structured data between relational databases |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108052681A true CN108052681A (en) | 2018-05-18 |
| CN108052681B CN108052681B (en) | 2020-05-26 |
Family
ID=62127506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810030156.2A Active CN108052681B (en) | 2018-01-12 | 2018-01-12 | Method and system for synchronizing structured data between relational databases |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108052681B (en) |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241183A (en) * | 2018-08-16 | 2019-01-18 | 武汉元鼎创天信息科技有限公司 | A kind of method of data synchronization and system based on socket communication |
| CN109359103A (en) * | 2018-09-04 | 2019-02-19 | 河南智云数据信息技术股份有限公司 | A kind of data aggregate cleaning method and system |
| CN109446262A (en) * | 2018-10-31 | 2019-03-08 | 成都四方伟业软件股份有限公司 | A kind of data assemblage method and device |
| CN109885532A (en) * | 2019-02-11 | 2019-06-14 | 中国银行股份有限公司 | A kind of transaction data standardized method and device |
| CN110197051A (en) * | 2019-06-13 | 2019-09-03 | 浪潮软件股份有限公司 | A kind of method, terminal and the computer readable storage medium of permission control |
| CN110297869A (en) * | 2019-05-30 | 2019-10-01 | 北京百度网讯科技有限公司 | A kind of AI Data Warehouse Platform and operating method |
| CN110413672A (en) * | 2019-07-03 | 2019-11-05 | 平安科技（深圳）有限公司 | Data automatically imported method, device and computer readable storage medium |
| CN110971685A (en) * | 2019-11-29 | 2020-04-07 | 腾讯科技（深圳）有限公司 | Content processing method, content processing device, computer equipment and storage medium |
| CN111061739A (en) * | 2019-12-17 | 2020-04-24 | 医渡云（北京）技术有限公司 | Method and device for warehousing massive medical data, electronic equipment and storage medium |
| CN111125065A (en) * | 2019-12-24 | 2020-05-08 | 阳光人寿保险股份有限公司 | Visual data synchronization method, system, terminal and computer readable storage medium |
| CN111143329A (en) * | 2019-12-27 | 2020-05-12 | 中国银联股份有限公司 | Data processing method and device |
| CN111159160A (en) * | 2019-12-31 | 2020-05-15 | 香港乐蜜有限公司 | Version rollback method and device, electronic equipment and storage medium |
| CN111475531A (en) * | 2020-04-12 | 2020-07-31 | 魏秋云 | Information analysis system based on student employment data |
| CN111694840A (en) * | 2020-04-29 | 2020-09-22 | 平安科技（深圳）有限公司 | Data synchronization method, device, server and storage medium |
| CN111694812A (en) * | 2020-05-06 | 2020-09-22 | 五八有限公司 | Data migration method and data migration device |
| CN111858632A (en) * | 2020-07-22 | 2020-10-30 | 浪潮云信息技术股份公司 | Storage method of incremental data of relational database based on NiFi |
| CN111881209A (en) * | 2020-06-29 | 2020-11-03 | 平安国际智慧城市科技股份有限公司 | Data synchronization method and device for heterogeneous database, electronic equipment and medium |
| CN111897877A (en) * | 2020-08-12 | 2020-11-06 | 浪潮软件股份有限公司 | High-performance and high-reliability data sharing system and method based on distributed thought |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
| CN112416907A (en) * | 2020-12-03 | 2021-02-26 | 厦门市美亚柏科信息股份有限公司 | Database table data importing and exporting method, terminal equipment and storage medium |
| CN112632176A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Interaction method and device for supervising submission database |
| WO2021103621A1 (en) * | 2019-11-25 | 2021-06-03 | 深圳壹账通智能科技有限公司 | Data construction method and apparatus, and computer device and storage medium |
| US20210200751A1 (en) * | 2019-12-31 | 2021-07-01 | Capital One Services, Llc | Monitoring and data validation of process log information imported from multiple diverse data sources |
| CN113392081A (en) * | 2021-06-10 | 2021-09-14 | 北京猿力未来科技有限公司 | Data processing system and method |
| CN113535857A (en) * | 2021-08-04 | 2021-10-22 | 阿波罗智联(北京)科技有限公司 | Data synchronization method and device |
| CN114595291A (en) * | 2022-05-10 | 2022-06-07 | 城云科技（中国）有限公司 | Collection task adjusting method and device based on database annotation |
| WO2022193893A1 (en) * | 2021-03-19 | 2022-09-22 | International Business Machines Corporation | Volatile database caching in a database accelerator |
| US11500733B2 (en) | 2021-03-19 | 2022-11-15 | International Business Machines Corporation | Volatile database caching in a database accelerator |
| CN116107816A (en) * | 2023-04-13 | 2023-05-12 | 山东捷瑞数字科技股份有限公司 | MYSQL database back-file cloud platform |
| US11797570B2 (en) | 2021-03-19 | 2023-10-24 | International Business Machines Corporation | Asynchronous persistency of replicated data changes in a database accelerator |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102129478A (en) * | 2011-04-26 | 2011-07-20 | 广州从兴电子开发有限公司 | Database synchronization method and system thereof |
| CN102542007A (en) * | 2011-12-13 | 2012-07-04 | 中国电子科技集团公司第十五研究所 | Method and system for synchronization of relational databases |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| CN103678532A (en) * | 2013-12-02 | 2014-03-26 | 中国移动（深圳）有限公司 | Alternation statement reverse analysis method, database alternating and backspacing method and database alternating and backspacing system |
| CN104516989A (en) * | 2015-01-26 | 2015-04-15 | 北京京东尚科信息技术有限公司 | Incremental data pushing system and method |
| CN106951536A (en) * | 2017-03-22 | 2017-07-14 | 努比亚技术有限公司 | Data method for transformation and system |
-
2018
- 2018-01-12 CN CN201810030156.2A patent/CN108052681B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102129478A (en) * | 2011-04-26 | 2011-07-20 | 广州从兴电子开发有限公司 | Database synchronization method and system thereof |
| CN102542007A (en) * | 2011-12-13 | 2012-07-04 | 中国电子科技集团公司第十五研究所 | Method and system for synchronization of relational databases |
| CN103067483A (en) * | 2012-12-25 | 2013-04-24 | 广东邮电职业技术学院 | Remote data increment synchronization method and device based on data package |
| CN103678532A (en) * | 2013-12-02 | 2014-03-26 | 中国移动（深圳）有限公司 | Alternation statement reverse analysis method, database alternating and backspacing method and database alternating and backspacing system |
| CN104516989A (en) * | 2015-01-26 | 2015-04-15 | 北京京东尚科信息技术有限公司 | Incremental data pushing system and method |
| CN106951536A (en) * | 2017-03-22 | 2017-07-14 | 努比亚技术有限公司 | Data method for transformation and system |
Non-Patent Citations (1)
| Title |
|---|
| 潘华 等: "《普通高等教育"十一五"规划教材 数据仓库与数据挖掘原理、工具及应用》", 31 December 2007, 中国电力出版社 * |
Cited By (39)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109241183B (en) * | 2018-08-16 | 2020-12-29 | 武汉元鼎创天信息科技有限公司 | Data synchronization method and system based on socket communication |
| CN109241183A (en) * | 2018-08-16 | 2019-01-18 | 武汉元鼎创天信息科技有限公司 | A kind of method of data synchronization and system based on socket communication |
| CN109359103A (en) * | 2018-09-04 | 2019-02-19 | 河南智云数据信息技术股份有限公司 | A kind of data aggregate cleaning method and system |
| CN109446262A (en) * | 2018-10-31 | 2019-03-08 | 成都四方伟业软件股份有限公司 | A kind of data assemblage method and device |
| CN109885532A (en) * | 2019-02-11 | 2019-06-14 | 中国银行股份有限公司 | A kind of transaction data standardized method and device |
| CN110297869A (en) * | 2019-05-30 | 2019-10-01 | 北京百度网讯科技有限公司 | A kind of AI Data Warehouse Platform and operating method |
| CN110197051A (en) * | 2019-06-13 | 2019-09-03 | 浪潮软件股份有限公司 | A kind of method, terminal and the computer readable storage medium of permission control |
| CN110413672A (en) * | 2019-07-03 | 2019-11-05 | 平安科技（深圳）有限公司 | Data automatically imported method, device and computer readable storage medium |
| CN110413672B (en) * | 2019-07-03 | 2023-09-19 | 平安科技（深圳）有限公司 | Automatic data importing method and device and computer readable storage medium |
| WO2021103621A1 (en) * | 2019-11-25 | 2021-06-03 | 深圳壹账通智能科技有限公司 | Data construction method and apparatus, and computer device and storage medium |
| CN110971685A (en) * | 2019-11-29 | 2020-04-07 | 腾讯科技（深圳）有限公司 | Content processing method, content processing device, computer equipment and storage medium |
| CN111061739A (en) * | 2019-12-17 | 2020-04-24 | 医渡云（北京）技术有限公司 | Method and device for warehousing massive medical data, electronic equipment and storage medium |
| CN111125065A (en) * | 2019-12-24 | 2020-05-08 | 阳光人寿保险股份有限公司 | Visual data synchronization method, system, terminal and computer readable storage medium |
| CN111125065B (en) * | 2019-12-24 | 2023-09-12 | 阳光人寿保险股份有限公司 | Visual data synchronization method, system, terminal and computer readable storage medium |
| CN111143329A (en) * | 2019-12-27 | 2020-05-12 | 中国银联股份有限公司 | Data processing method and device |
| CN111143329B (en) * | 2019-12-27 | 2024-02-13 | 中国银联股份有限公司 | Data processing method and device |
| CN111159160B (en) * | 2019-12-31 | 2023-06-20 | 卓米私人有限公司 | Version rollback method and device, electronic equipment and storage medium |
| CN111159160A (en) * | 2019-12-31 | 2020-05-15 | 香港乐蜜有限公司 | Version rollback method and device, electronic equipment and storage medium |
| US20210200751A1 (en) * | 2019-12-31 | 2021-07-01 | Capital One Services, Llc | Monitoring and data validation of process log information imported from multiple diverse data sources |
| CN111475531A (en) * | 2020-04-12 | 2020-07-31 | 魏秋云 | Information analysis system based on student employment data |
| CN111694840A (en) * | 2020-04-29 | 2020-09-22 | 平安科技（深圳）有限公司 | Data synchronization method, device, server and storage medium |
| WO2021218143A1 (en) * | 2020-04-29 | 2021-11-04 | 平安科技（深圳）有限公司 | Data synchronization method and apparatus, and server and storage medium |
| CN111694840B (en) * | 2020-04-29 | 2023-05-30 | 平安科技（深圳）有限公司 | Data synchronization method, device, server and storage medium |
| CN111694812A (en) * | 2020-05-06 | 2020-09-22 | 五八有限公司 | Data migration method and data migration device |
| CN111881209A (en) * | 2020-06-29 | 2020-11-03 | 平安国际智慧城市科技股份有限公司 | Data synchronization method and device for heterogeneous database, electronic equipment and medium |
| CN111858632B (en) * | 2020-07-22 | 2024-02-20 | 浪潮云信息技术股份公司 | NiFi-based relational database incremental data warehousing method |
| CN111858632A (en) * | 2020-07-22 | 2020-10-30 | 浪潮云信息技术股份公司 | Storage method of incremental data of relational database based on NiFi |
| CN111897877B (en) * | 2020-08-12 | 2024-03-26 | 浪潮软件股份有限公司 | High-performance high-reliability data sharing system and method based on distributed ideas |
| CN111897877A (en) * | 2020-08-12 | 2020-11-06 | 浪潮软件股份有限公司 | High-performance and high-reliability data sharing system and method based on distributed thought |
| CN112364101A (en) * | 2020-11-11 | 2021-02-12 | 深圳前海微众银行股份有限公司 | Data synchronization method and device, terminal equipment and medium |
| CN112416907A (en) * | 2020-12-03 | 2021-02-26 | 厦门市美亚柏科信息股份有限公司 | Database table data importing and exporting method, terminal equipment and storage medium |
| CN112632176A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Interaction method and device for supervising submission database |
| US11500733B2 (en) | 2021-03-19 | 2022-11-15 | International Business Machines Corporation | Volatile database caching in a database accelerator |
| WO2022193893A1 (en) * | 2021-03-19 | 2022-09-22 | International Business Machines Corporation | Volatile database caching in a database accelerator |
| US11797570B2 (en) | 2021-03-19 | 2023-10-24 | International Business Machines Corporation | Asynchronous persistency of replicated data changes in a database accelerator |
| CN113392081A (en) * | 2021-06-10 | 2021-09-14 | 北京猿力未来科技有限公司 | Data processing system and method |
| CN113535857A (en) * | 2021-08-04 | 2021-10-22 | 阿波罗智联(北京)科技有限公司 | Data synchronization method and device |
| CN114595291A (en) * | 2022-05-10 | 2022-06-07 | 城云科技（中国）有限公司 | Collection task adjusting method and device based on database annotation |
| CN116107816A (en) * | 2023-04-13 | 2023-05-12 | 山东捷瑞数字科技股份有限公司 | MYSQL database back-file cloud platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108052681B (en) | 2020-05-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108052681A (en) | The synchronous method and system of structural data between a kind of relevant database | |
| CN103440273B (en) | A kind of data cross-platform migration method and device | |
| CN110175213A (en) | A kind of oracle database synchronization system and method based on SCN mode | |
| CN106528129A (en) | Web application interface generation system and method | |
| CN104104738A (en) | FTP-based (file transfer protocol-based) data exchange system | |
| CN104573100A (en) | Step-by-step database synchronization method with autoincrement identifications | |
| CN106227800A (en) | The storage method of the big data of a kind of highlights correlations and management system | |
| JP2018514044A (en) | System and method for querying a data source | |
| CN101000677A (en) | Matching method and system of marketing business recommended | |
| CN109213820B (en) | Method for realizing fusion use of multiple types of databases | |
| CN102508908A (en) | Method for acquiring subordinate financial business data and system for acquiring subordinate financial business data | |
| US11176173B2 (en) | Arrangement for enriching data stream in a communications network and related method | |
| CN101968793A (en) | Method and system for checking on basis of disparate data source data | |
| CN113992769B (en) | Industrial Internet information exchange method | |
| CN111327446B (en) | Configuration data processing method, software defined network device, system and storage medium | |
| CN104767795A (en) | LTE MRO data statistical method and system based on HADOOP | |
| CN107622055A (en) | A kind of quick method for realizing data, services issue | |
| CN105630500A (en) | CIM object model for data processing and programming frame thereof | |
| CN107688611A (en) | A kind of Redis key assignments management system and method based on saltstack | |
| CN104504030B (en) | A kind of indexing means towards power dispatching automation magnanimity message | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN101170436B (en) | A method for managing template in network management system | |
| CN105550351B (en) | The extemporaneous inquiry system of passenger's run-length data and method | |
| CN105933396B (en) | A kind of each canalization interface of collaboration realizes the method and system of long flow | |
| CN108846002B (en) | Label real-time updating method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210406 Address after: No.26 Fucheng Road, Haidian District, Beijing 100142 Patentee after: MILITARY SCIENCE INFORMATION RESEARCH CENTER OF MILITARY ACADEMY OF THE CHINESE PLA Address before: 100142 courtyard 26, Fucheng Road, Haidian District, Beijing Patentee before: Mao Bin Patentee before: MILITARY SCIENCE INFORMATION RESEARCH CENTER OF MILITARY ACADEMY OF THE CHINESE PLA |