CN110209730A - Change synchronous method, device, computer equipment and the computer storage medium of data - Google Patents

Change synchronous method, device, computer equipment and the computer storage medium of data Download PDF

Info

Publication number
CN110209730A
CN110209730A CN201910337189.6A CN201910337189A CN110209730A CN 110209730 A CN110209730 A CN 110209730A CN 201910337189 A CN201910337189 A CN 201910337189A CN 110209730 A CN110209730 A CN 110209730A
Authority
CN
China
Prior art keywords
table structure
change
data
business side
business
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910337189.6A
Other languages
Chinese (zh)
Inventor
张辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
OneConnect Smart Technology Co Ltd
Original Assignee
OneConnect Smart Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by OneConnect Smart Technology Co Ltd filed Critical OneConnect Smart Technology Co Ltd
Priority to CN201910337189.6A priority Critical patent/CN110209730A/en
Publication of CN110209730A publication Critical patent/CN110209730A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • G06F16/254Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

This application discloses it is a kind of change data synchronous method, device and computer storage medium, be related to cloud storage technical field, may be implemented target database automatically kept with table structure in business party database it is synchronous.The described method includes: obtaining the table structure that there is variation in unit synchronization time in each business party database, the change table structure of each business side is obtained;The change table structure of each business side is compared with the table structure of target database, filters out the change table structure for comparing inconsistent each business side;The change table structure for comparing inconsistent each business side is parsed, and the list of modification structure mapping for comparing inconsistent each business side is generated into the change table structure of the target database into the table structure of target database;The data in the change table structure for comparing inconsistent each business side are extracted as change data, the change data are synchronized in the change table structure in target database according to pre-set isochronous schedules parameter.

Description

Change synchronous method, device, computer equipment and the computer storage medium of data
Technical field
The present invention relates to cloud storage technical fields, set particularly with regard to synchronous method, device, the computer of change data Standby and computer storage medium.
Background technique
With gradually increasing for corporate business, the data of company's items business are collected by establishing service database And maintenance, when corporate HQ obtains the service data information of each branch company or big number does to the business of each company in acquisition side When according to acquisition, need the database of corporate HQ or acquisition side is synchronous with the holding of the service database of each company.
Currently, when realizing that each business party database is synchronous with the data structure of target database, due to business datum Table in library is numerous, and service line often will increase the structure of table or change table, and acquisition side usually needs and the artificial ditch in business side It is logical, the Data Definition Language of table in service database is provided by business side, equally executes the number of table in target database later According to definitional language, keep service database consistent with the data structure in target database.However, the table in business party database When number is numerous, business side is often difficult to provide newest table structure, and the data of accurate table are obtained from each of business side line Definitional language needs are largely linked up, and are spent very big human cost, are easy to cause the data definition of the table artificially provided Language and really inconsistent, causes business party database asynchronous with the data structure of target database, causes data same Shortage of data during step.
Summary of the invention
In view of this, the present invention provides a kind of synchronous method, device, computer equipment and computers for changing data to deposit Storage media, main purpose are the data structure of business party database and target database caused by solving artificially to link up at present not Synchronous problem.
According to the present invention on one side, a kind of synchronous method for changing data is provided, this method comprises:
The table structure that there is variation in unit synchronization time in each business party database is obtained, each business side is obtained Change table structure;
The change table structure of each business side is compared with the table structure of target database, filters out comparison not The change table structure of consistent each business side;
The change table structure for comparing inconsistent each business side is parsed, and compares inconsistent each industry for described The list of modification structure mapping of business side generates the change table structure of the target database into the table structure of target database;
The data in the change table structure for comparing inconsistent each business side are extracted as change data, according to pre- The change data are synchronized in the change table structure in target database by the isochronous schedules parameter being first arranged.
Further, described to obtain the table structure that there is variation in unit synchronization time in each business party database, it obtains Change table structure to each business side includes:
Table structure in timing acquiring unit synchronization time in each business party database, obtains the table knot of each business side Structure;
Starting inspection scheduler task, detects the change in the table structure of each business side according to the inspection scheduler task More content obtains the change table structure of each business side.
Further, the table structure by the change table structure of each business side and target database compares Right, the change table structure for filtering out the inconsistent each business side of comparison includes:
The table knot of the metadata and the target database in the change table structure of each business side is extracted respectively Metadata in structure;
By the member in the table structure of metadata and the target database in the change table structure of each business side Data are compared, and filter out the change table structure for comparing inconsistent each business side.
Further, the metadata by the change table structure of each business side and the target database Metadata in table structure is compared, and the change table structure for filtering out the inconsistent each business side of comparison includes:
By the table of metadata corresponding logical construction and the target database in the change table structure of the business side The corresponding logical construction of metadata in structure is compared, and filters out the list of modification of the inconsistent each business side of logical construction Structure;
By the table of metadata corresponding storage content and the target database in the change table structure of the business side The corresponding storage content of metadata in structure is compared, and filters out the list of modification of the inconsistent each business side of storage content Structure.
Further, the parsing change table structure for comparing inconsistent each business side, and by the comparison The list of modification structure mapping of inconsistent each business side generates the target database into the table structure of target database Changing table structure includes:
According to the change table structure for comparing inconsistent each business side, the corresponding number of the change table structure is generated According to definition statement;
By executing the corresponding data definition statement of the change table structure in the target database, by the comparison The list of modification structure mapping of inconsistent each business side generates the target database into the table structure of target database Change table structure.
Further, the change data are synchronized to by target data according to pre-set isochronous schedules parameter described Before in change table structure in library, which comprises
According to the size of the change data corresponding number, the data volume rank of the change data is determined;
According to the data volume rank of the change data, the corresponding isochronous schedules parameter of the change data is set.
Further, the isochronous schedules parameter include synchronization time parameter, isochronous schedules time level parameters and same Node weights parameter is walked, the corresponding same step of the change data is arranged in the data volume rank according to the change data Spending parameter includes:
The time required for the change data complete synchronous task is estimated according to the data volume rank of the change data, Parameter synchronization time of the change data is set;
When estimating the unit that the change data are dispatched in synchronous task according to the data volume rank of the change data Between, the isochronous schedules time level parameters of the change data are set;
The change data are estimated according to the data volume rank of the change data to need to open synchronization in synchronous task The synchronization node weight parameter of the change data is arranged in number of nodes.
According to the present invention on the other hand, a kind of synchronizing device for changing data is provided, described device includes:
Acquiring unit is obtained for obtaining the table structure that there is variation in unit synchronization time in each business party database To the change table structure of each business side;
Screening unit, for comparing the table structure of the change table structure of each business side and target database It is right, filter out the change table structure for comparing inconsistent each business side;
Generation unit, for parsing the change table structure for comparing inconsistent each business side, and by the comparison The list of modification structure mapping of inconsistent each business side generates the target database into the table structure of target database Change table structure;
Synchronization unit, for extracting the data in the change table structure for comparing inconsistent each business side as change The change data are synchronized to the change table structure in target database according to pre-set isochronous schedules parameter by more data In.
Further, the acquiring unit includes:
Module is obtained, for the table structure in each business party database in timing acquiring unit synchronization time, is obtained each The table structure of a business side;
Detection module detects each business side according to the inspection scheduler task for starting inspection scheduler task Table structure in changed content, obtain the change table structure of each business side.
Further, the screening unit includes:
Extraction module, the metadata and the target in change table structure for extracting each business side respectively Metadata in the table structure of database;
Screening module, for by the metadata and the target database in the change table structure of each business side Metadata in table structure is compared, and filters out the change table structure for comparing inconsistent each business side.
Further, the screening module, it is corresponding specifically for the metadata in the change table structure by the business side Logical construction be compared with the corresponding logical construction of metadata in the table structure of the target database, filter out logic The change table structure of the inconsistent each business side of structure;
The screening module is specifically also used in the corresponding storage of metadata in the change table structure by the business side Appearance is compared with the corresponding storage content of metadata in the table structure of the target database, and it is different to filter out storage content The change table structure of each business side caused.
Further, the generation unit includes:
First generation module, for according to the change table structure for comparing inconsistent each business side, described in generation Change the corresponding data definition statement of table structure;
Second generation module, for fixed by executing the corresponding data of the change table structure in the target database Adopted sentence, it is raw by the list of modification structure mapping for comparing inconsistent each business side into the table structure of target database At the change table structure of the target database.
Further, described device further include:
Determination unit, for the change data to be synchronized to target according to pre-set isochronous schedules parameter described Before in change table structure in database, according to the size of the change data corresponding number, the change data are determined Data volume rank;
The corresponding synchronization of the change data is arranged for the data volume rank according to the change data in setting unit Scheduling parameter.
Further, the isochronous schedules parameter include synchronization time parameter, isochronous schedules time level parameters and same Node weights parameter is walked, the setting unit includes:
First setting unit is completed to synchronize for estimating the change data according to the data volume rank of the change data Parameter synchronization time of the change data is arranged in the time that required by task is wanted;
Second setting unit is appointed for estimating the change data according to the data volume rank of the change data synchronous The isochronous schedules time level parameters of the change data are arranged in the unit time dispatched in business;
Third setting unit is appointed for estimating the change data according to the data volume rank of the change data synchronous It needs to open synchronization node quantity in business, the synchronization node weight parameter of the change data is set.
Another aspect according to the present invention provides a kind of computer equipment, including memory and processor, the storage Device is stored with computer program, and the processor realizes the step of the synchronous method of change data when executing the computer program Suddenly.
Another aspect according to the present invention provides a kind of computer storage medium, is stored thereon with computer program, institute The step of stating the synchronous method that change data are realized when computer program is executed by processor.
By above-mentioned technical proposal, the present invention provides a kind of synchronous method and device for changing data, each by obtaining The change table structure of business side, and the change table structure of each business side is compared with the table structure of target database, it sieves The change table structure for comparing inconsistent each business side is selected, the data in the presence of change can be found in time, further in mesh It marks and generates change table structure in database, so that the data extracted in the change table structure of each business side are used as change data, Change data are synchronized in the change table structure in target database, guarantee that the change data of business side can be synchronized in time In target database.The synchronous of change data is realized by way of artificially linking up with business side with acquisition side in the prior art Method is compared, and the present invention becomes by the table structure in timing acquiring business party database, while by existing in business party database Table structure more is compared with the table structure in target database, and the change table structure for comparing inconsistent business side is reflected It is mapped in target database, it is synchronous to realize that target database is automatically kept with table structure in business party database, reduce a large amount of Artificial communication, therefore, change data synchronizing process in, avoid target database and business as caused by artificially linking up The inconsistent problem of party database improves the synchronous efficiency of change data.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings:
Fig. 1 shows a kind of synchronous method flow diagram for changing data provided in an embodiment of the present invention;
Fig. 2 shows the synchronous method flow diagrams of another change data provided in an embodiment of the present invention;
Fig. 3 shows a kind of structural schematic diagram of synchronizing device for changing data provided in an embodiment of the present invention;
Fig. 4 shows the structural schematic diagram of the synchronizing device of another change data provided in an embodiment of the present invention.
Specific embodiment
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawing Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure It is fully disclosed to those skilled in the art.
The embodiment of the invention provides a kind of synchronous method for changing data, and target database and business number formulary may be implemented Synchronization is automatically kept according to table structure in library, as shown in Figure 1, this method comprises:
101, the table structure that there is variation in unit synchronization time in each business party database is obtained, each business is obtained The change table structure of side.
Wherein, the data stored in business party database can be the number that database generates during storage or operation According to for example, the production data that server generates, the user information data etc. of client input, can also be to record user in platform The daily record data of behavior, for example, user browses the data of webpage, user buys the data etc. of commodity, here without limiting.
It is understood that in order to guarantee making full use of for resource data, each business side can be by number after generating data According to be synchronized to department that other need data resource perhaps business side and in each business side to other departments or business side It carries out in data synchronization process, business side, which changes data, belongs to normal phenomenon, data while business side changes data The table structure in library can also generate change, further obtain the table that there is variation in unit synchronization time in each business party database Structure obtains the change table structure of each business side.
It, specifically can be by interval of the table in unit monitoring business synchronization time party database for the embodiment of the present invention Structure judges then to illustrate business once discovery table structure has variation with the presence or absence of variation in the table structure of business party database Side changes data, further obtains in business party database and there is the table structure of variation to get the change of business side is arrived Table structure.
It, can be by dispatch server to being adopted it is understood that for the ease of being operated to business party database The information of the business party database of collection is configured, and the information of business party database includes the IP address of business party database, number According to the type etc. in library, after the completion of dispatch server is to the information configuration of business party database, believe as the mark of business side Breath, to be realized by the identification information of business side between each business side in the synchronizing process of subsequent change data Communication.
102, the change table structure of each business side is compared with the table structure of target database, filters out ratio To the change table structure of inconsistent each business side.
It should be noted that the change data of business side may be simultaneously when there is change table structure in business party database It is not synchronized in target database, or changing the corresponding synchrodata of table structure in synchronizing process may in synchronizing process There can be missing, lead in data synchronization process that there are problems.
To with the embodiment of the present invention, if the change table structure of each business side is consistent with the table structure of target database, Then illustrate that there is no change data for business side, so table structure does not change;If the change table structure of each business side It is inconsistent with the table structure of target database, then illustrate that business number formulary changes data according to presence, and table structure changes, by This, can find the table structure changed in business party database in time, avoid when the table number in business party database is numerous, example Such as, when having up to a hundred or even upper thousand sheets table in business party database, by artificially linking up the table structure omitted and updated, to make At the asynchronous of business party database and target database.
103, parse the change table structure for comparing inconsistent each business side, and by it is described compare it is inconsistent each The list of modification structure mapping of a business side generates the list of modification knot of the target database into the table structure of target database Structure.
Wherein, the table structure that there is variation in business party database includes the increase and business of table in business party database The change of table structure in party database.
Target data is illustrated for comparing the change table structure of inconsistent each business side for the embodiment of the present invention In library and the table structure of the business side is not present, if it is same using original table structure to likely result in data in target database Data are generated in step failure or data synchronization process to omit, and the change table structure of inconsistent business side is compared by parsing, And by the list of modification structure mapping into the table structure of target database, the change table structure of target database is generated, due to mesh The table structure of mark database is also mapped as change table structure, consequently facilitating changing the synchronization of data in each business side, guarantees to become The synchronous accuracy of more data.
104, the data in the change table structure for comparing inconsistent each business side are extracted as change data, root The change data are synchronized in the change table structure in target database according to pre-set isochronous schedules parameter.
Table structure for the embodiment of the present invention, in the table structure and target database in the business party database of extraction Table when inconsistent, i.e., when business party database is asynchronous with the table structure in target database, in corresponding service party database The corresponding data of structure are also further to extract the data in business side in table structure in the presence of change and be used as change data, from And by business number formulary according to demand business side is supplied to, realize that carrying out data to change data synchronizes.
Wherein, pre-set isochronous schedules parameter may include synchronization time parameter, isochronous schedules time level parameters And synchronization node weight parameter, parameter synchronization time is to estimate completion change data to synchronize the required time, for example, configuration The synchronization of change data is completed in 1 hour, isochronous schedules time rank is that the unit that change data are dispatched in synchronizing process is same The time is walked, for example, 5 minutes ranks, 10 minutes ranks, 30 minutes ranks, 1 hour rank, 3 hours ranks and 1 day rank, together Step node weights parameter is the number of nodes for changing data and needing to open simultaneously in synchronizing process, for example, opening 5 machines simultaneously It is synchronous that device changes data.Further according to preconfigured isochronous schedules parameter, change data are synchronized to target data In library.
The embodiment of the present invention provides a kind of synchronous method for changing data, by the list of modification knot for obtaining each business side Structure, and the change table structure of each business side is compared with the table structure of target database, it is inconsistent to filter out comparison The change table structure of each business side can find the data in the presence of change in time, further generate and become in target database Change data are synchronized to by more table structure so that the data extracted in the change table structure of each business side are used as change data In change table structure in target database, guarantee that the change data of business side can be synchronized in target database in time.With Acquisition side in such a way that business side is artificially linked up compared with realizing the synchronous method of change data in the prior art, the present invention By the table structure in timing acquiring business party database, while the table structure and target that will there is change in business party database Table structure in database is compared, and will compare the list of modification structure mapping of inconsistent business side to target database In, it is synchronous to realize that target database is automatically kept with table structure in business party database, reduces a large amount of artificial communication, because This avoids the target database as caused by artificially linking up and business party database is different in the synchronizing process of change data The problem of cause, improves the synchronous efficiency of change data.
The embodiment of the invention provides the synchronous method of another change data, can find in time in business party database Change table structure, guarantee business party database and target database consistency, the missing of synchrodata is avoided, such as Fig. 2 institute Show, which comprises
201, the table structure in timing acquiring unit synchronization time in each business party database, obtains each business side Table structure.
For the embodiment of the present invention, since the table number that business side usually increases service line or business party database is more, In order to find the variation of table structure in each business party database in time, can specifically be believed according to the mark of business party database Breath connects business party database, the database of each business side is tracked and acquired, and extracts the table in business party database Structure, for example, can (Extract-Transform-Load/ data pick-up-interaction is converted-is loaded onto destination by ETL Process) timing acquiring business party database table structure, ETL can be realized by way of by tool, for example, passing through ETL tool in Kettle software is tracked and is acquired to business party database, first to the business that will be tracked and acquire Party database information is configured, type and IP address including business party database etc., later to ETL track and acquire when Between configured, for example, the shorter time can be 10 minutes perhaps half an hour, that is, each 10 minutes or half an hours It is primary to the table structure acquisition of business party database.It, can be to each table in each business party database in tracking and acquisition Set is tracked and is acquired, and the table structure of each table set in business party database is extracted.
For example, extracting the table structure of each business number formulary evidence by the ETL tool in Kettle software, matched by input table The information of purchase of property business party database, type Oracle, SQL server, MySQL including business party database etc., selection driving Type, the IP address of business party database, Instance Name, port numbers and the user name password of business party database.
202, starting inspection scheduler task, is detected in the table structure of each business side according to the inspection scheduler task Changed content, obtain the change table structure of each business side.
After business side uploads data synchronous task, once business side is modified synchrodata, it can all lead to business The table structure of party database changes, and the data needs of variation artificially go to adjust manually, changes table structures if there is a large amount of, Easily there is loss of data.
The embodiment of the present invention periodically monitors the table knot of each business side in the unit time by starting inspection scheduler task Changed content in structure, which can periodically go to check the table structure in each business party database, for example, can Interval 1 hour, 2 hours or 1 day etc. is arranged, here to interval time without limiting.
203, metadata in the change table structure of each business side and the target database are extracted respectively Metadata in table structure.
For the embodiment of the present invention, when the table structure in each business side changes, illustrate the change of each business side Metadata in table structure equally changes, metadata in change table structure by extracting each business side respectively and Metadata in the table structure of target database enables and is drawn into variation in time when business side's table structure changes Metadata, and the metadata of variation is handled, solve the problems, such as that business side and table structure in target database are inconsistent.
It 204, will be in the table structure of metadata and the target database in the change table structure of each business side Metadata be compared, filter out the change table structure for comparing inconsistent each business side.
For the embodiment of the present invention, acquisition can be timed to each table structure in business party database by ETL, And the metadata in table structure is collected, which deposits more particularly to the corresponding logical construction of metadata is corresponding with metadata Storage structure likewise, being acquisition surely to each table structure in target database by ETL, and collects in table structure Metadata, acquisition is further included in an inventory to the metadata in business side's table structure, by collected number of targets It is included in another inventory according to the metadata in the table structure in library, by the metadata and mesh in two inventories in the table structure of business side The metadata marked in the table structure of database compares, can be first by the member in the change table structure of business side when comparison The corresponding logical construction of data is compared with the corresponding logical construction of metadata in the table structure of target database, filters out The change table structure of the inconsistent each business side of logical construction;Then the metadata in the change table structure of business side is corresponding Storage organization be compared with the corresponding storage organization of metadata in the table structure of target database, filter out storage organization The change table structure of inconsistent each business side;The list of modification knot of the inconsistent each business side of further summarization logic structure The change table structure of structure and the inconsistent each business side of storage organization obtains the list of modification for comparing inconsistent each business side Structure.
Specifically during comparison metadata corresponding logic content, the change table structure of business side can be compared first In metadata correspond to the quantity of the corresponding table of the metadata in the quantity of table and the table structure of target database, secondly compare business Metadata in the change table structure of side corresponds to the word of the corresponding table of the metadata in the field of table and the table structure of target database Section, for example, the table structure A of business party database is compared with the table structure A ' of target database, by business party database There are 50 tables in table structure A, and there are 45 tables in the table structure A ' of target database, then target database and business number formulary at this time Inconsistent according to the table structure in library, 5 tables for needing to change in the table structure A by business party database are synchronized to target database In table structure A '.
It should be noted that the table structure of metadata and target database in the change table structure by each business side In metadata be compared during, can also accordingly compare the corresponding table of metadata in the change table structure of business side Table mark corresponding with the metadata in the table structure of target database, table name etc. are identified, here without limiting.
205, according to the change table structure for comparing inconsistent each business side, it is corresponding to generate the change table structure Data definition statement.
For the embodiment of the present invention, the change table structure of each business side can for increased newly in business party database table or Newly-increased table structure, specifically can pull an inventory for the change table structure for comparing inconsistent each business side, pass through operation Simple structured query language editing machine is provided in software Kettle, software Kettle, table in database may be implemented Index or modification field are deleted in creation, can support most of data definition statement, such as: creation table, change table, wound It indexes, create the orders such as sequence and can be run by structured query language editor window, compared in input inconsistent Each business side change table structure after, the corresponding data definition language of change table structure can be automatically generated in Kettle Sentence.
It should be noted that the change table structure due to different business side is different, the corresponding number of change table structure of generation May be different according to definition statement, in order to support the service database of different business line mutually, may be implemented to supervise automatically Control, can to the data type in different business database use identical definition, for example, in different business database data essence Spend it is identical, double data type use identical length.
It 206, will be described by executing the corresponding data definition statement of the change table structure in the target database The list of modification structure mapping of inconsistent each business side is compared into the table structure of target database, generates the target data The change table structure in library.
It should be noted that the change table structure due to different business side is different, the corresponding number of change table structure of generation May be different according to definition statement, in order to support the service database of different business line mutually, may be implemented to supervise automatically Control, can to the data type in different business database use identical definition, for example, in different business database data essence Spend it is identical, double data type use identical length.
For the embodiment of the present invention, generating the corresponding data definition statement of change table structure may be implemented various data manipulations Instruction can will be compared inconsistent each by executing the corresponding data definition statement of change table structure in target database The list of modification structure mapping of a business side is into the table structure of target database, if table structure changes in business party database Become, after executing the corresponding data definition statement of change table structure, table structure can also occur accordingly equally in target database Change, to generate the change table structure of target database.Hereby it is achieved that business party database and target data database table structure Automatic synchronization, reduce it is artificial link up cost, improve the synchronous efficiency of change data, while also avoiding human factor and making At target database and the nonsynchronous problem of business party database table structure.
207, the data in the change table structure for comparing inconsistent each business side are extracted as change data, root According to the size of the change data corresponding number, the data volume rank of the change data is determined.
For the embodiment of the present invention, since business side is after uploading data synchronous task, for the ease of business side to change The synchronous task of more data is managed, and business side can in advance count the corresponding data volume of change data, so that it is determined that Change the data volume rank of data.
For example, the data that statistical data amount is greater than 1000 can be divided into level-one, illustrates that data volume is more, usually need The time to be synchronized is longer, and the data that statistical data amount is 500-1000 item are divided into second level, illustrates that data volume is medium, needs The synchronous time, data of the statistical data amount less than 5000 placed in the middle were divided into three-level, illustrated that data volume is less, it usually needs same The time of step is less, can divide more without limiting with specific reference to actual conditions to the other division rule of data magnitude here More data volume ranks.
208, according to the data volume rank of the change data, the corresponding isochronous schedules parameter of the change data is set.
For the embodiment of the present invention, change the corresponding isochronous schedules parameter of data may include synchronization time parameter, synchronization Scheduling time level parameters and synchronization node weight parameter, parameter synchronization time are to estimate change data to complete required for synchronizing Time, for example, configuration 1 hour in complete change data synchronization, isochronous schedules time level parameters be change data same The unit time dispatched during step, for example, 5 minutes ranks, 10 minutes ranks, 30 minutes ranks, 1 hour rank, 3 hours grades Not and 1 day rank, synchronization node weight parameter are the number of nodes for changing data and needing to open simultaneously in synchronizing process, example Such as, while 5 servers of unlatching or 10 servers carry out data and synchronize.
For the embodiment of the present invention, with specific reference to the data volume rank of change data, the corresponding synchronization of setting change data Required for scheduling parameter includes the following steps: that the data volume rank according to change data estimates change data completion synchronous task Time, parameter synchronization time of setting change data;Change data are estimated according to the data volume rank of change data to appoint synchronous The unit time dispatched in business, the isochronous schedules time level parameters of setting change data;According to the data magnitude of change data Change data are not estimated needs to open synchronization node quantity, the synchronization node weight ginseng of setting change data in synchronous task Number.
It is illustrated with the isochronous schedules time level parameters citing in isochronous schedules parameter, is specifically being directed to different data During the corresponding isochronous schedules time level parameters of the other setting change data of magnitude, if the data magnitude of change data It Wei level-one, it is believed that the data volume for changing data is larger, needs to expend that complete data for a long time synchronous, it is possible to will 1 hour rank or 3 hours ranks etc. can be set in the isochronous schedules time rank of change data, if the data of change data Magnitude is not second level, it is believed that the data volume for changing data is placed in the middle, and it is synchronous to complete data to need to expend medium while, so 1 hour rank or 30 minutes ranks etc. can be set by the isochronous schedules time rank for changing data, if change data Data volume rank be three-level, it is believed that the data volume for changing data is less, needs the short time to can be completed data synchronous, institute 5 minutes ranks or 10 minutes ranks etc. can be set by the isochronous schedules time rank for changing data.
209, the change data are synchronized to by the change in target database according to pre-set isochronous schedules parameter In table structure.
Since after receiving data synchronic command, business side can count the synchronizing information of change data in advance, and such as where is statistics A little tables of data needs synchronize, and change data correspond to the link information of target database, and synchronous deadline limitation Etc., isochronous schedules parameter is further arranged according to the obtained synchronizing information of statistics, i.e., completion data synchronize the required time, Isochronous schedules time rank and synchronization node weight parameter business side, for example, business side requires to complete 15 data in 3 days The data synchronous task of table, here be arranged isochronous schedules parameter be synchronization time parameter be 3 days, isochronous schedules time level parameters For 1 day rank, synchronization node weight parameter was 5 servers, and every server distributes at least one tables of data daily and completes data Synchronous task is realized further according to pre-set isochronous schedules parameter the change data in business side being synchronized to number of targets According to library.
Further, the specific implementation as Fig. 1 the method, the embodiment of the invention provides a kind of the same of change data Device is walked, as shown in figure 3, described device includes: acquiring unit 31, screening unit 32, generation unit 33, synchronization unit 34.
Acquiring unit 31 can be used for obtaining the table knot that there is variation in unit synchronization time in each business party database Structure obtains the change table structure of each business side;
Screening unit 32, can be used for by the table structure of the change table structure of each business side and target database into Row compares, and filters out the change table structure for comparing inconsistent each business side;
Generation unit 33, can be used for parsing the change table structure for comparing inconsistent each business side, and by institute The list of modification structure mapping for comparing inconsistent each business side is stated into the table structure of target database, generates the number of targets According to the change table structure in library;
Synchronization unit 34 can be used for extracting the data in the change table structure for comparing inconsistent each business side As change data, the change data are synchronized to by the change in target database according to pre-set isochronous schedules parameter In table structure.
A kind of synchronizing device changing data provided by the invention, by obtaining the change table structure of each business side, and The change table structure of each business side is compared with the table structure of target database, filters out and compares inconsistent each industry The change table structure of business side can find the data in the presence of change in time, and list of modification knot is further generated in target database Change data are synchronized to number of targets so that the data extracted in the change table structure of each business side are used as change data by structure According in the change table structure in library, the change data of guarantee business side can be synchronized in target database in time.With existing skill Compared with realizing the synchronous method of change data in such a way that business side is artificially linked up, it is fixed that the present invention passes through for acquisition side in art When capturing service party database in table structure, while will in business party database exist change table structure and target database In table structure be compared, and the list of modification structure mapping of inconsistent business side will be compared into target database, realized Target database automatically kept with table structure in business party database it is synchronous, reduce it is a large amount of artificial link up, therefore, changing In the synchronizing process of data, the target database as caused by artificially linking up and the inconsistent problem of business party database are avoided, Improve the synchronous efficiency of change data.
As the further explanation for the synchronizing device for changing data shown in Fig. 3, Fig. 4 is another according to embodiments of the present invention The structural schematic diagram of the synchronizing device of kind change data, as shown in figure 4, described device further include:
Determination unit 35 can be used for according to pre-set isochronous schedules parameter that the change data are synchronous described Before in change table structure into target database, according to the size of the change data corresponding number, the change is determined The data volume rank of data;
Setting unit 36 can be used for that it is corresponding that the change data are arranged according to the data volume rank of the change data Isochronous schedules parameter.
Further, the isochronous schedules parameter include synchronization time parameter, isochronous schedules time level parameters and same Node weights parameter is walked, the setting unit 36 includes:
First setting unit 361 can be used for estimating the change data according to the data volume rank of the change data The time required for synchronous task is completed, parameter synchronization time of the change data is set;
Second setting unit 362 can be used for estimating the change data according to the data volume rank of the change data The isochronous schedules time level parameters of the change data are arranged in the unit time dispatched in synchronous task;
Third setting unit 363 can be used for estimating the change data according to the data volume rank of the change data It needs to open synchronization node quantity in synchronous task, the synchronization node weight parameter of the change data is set.
Further, the acquiring unit 31 includes:
Module 311 is obtained, can be used for the table structure in timing acquiring unit synchronization time in each business party database, Obtain the table structure of each business side;
Detection module 312 can be used for starting inspection scheduler task, be detected according to the inspection scheduler task described each Changed content in the table structure of business side obtains the change table structure of each business side.
Further, the screening unit 32 includes:
Extraction module 321, can be used for extracting respectively metadata in the change table structure of each business side and Metadata in the table structure of the target database;
Screening module 322 can be used for the metadata and the target in the change table structure of each business side Metadata in the table structure of database is compared, and filters out the change table structure for comparing inconsistent each business side.
Further, the screening module 322 specifically can be used for first number in the change table structure by the business side It is compared, screens with the corresponding logical construction of metadata in the table structure of the target database according to corresponding logical construction The change table structure of the inconsistent each business side of logical construction out;
The screening module 322, the metadata that specifically can be also used in the change table structure by the business side are corresponding Storage content be compared with the corresponding storage content of metadata in the table structure of the target database, filter out storage The change table structure of the inconsistent each business side of content.
Further, the generation unit 33 includes:
First generation module 331 can be used for according to the change table structure for comparing inconsistent each business side, raw At the corresponding data definition statement of the change table structure;
Second generation module 332 can be used for corresponding by executing the change table structure in the target database Data definition statement, by the table knot of the list of modification structure mapping for comparing inconsistent each business side to target database In structure, the change table structure of the target database is generated.
It should be noted that its of each functional unit involved by a kind of synchronizing device for changing data provided in this embodiment He accordingly describes, can be with reference to the corresponding description in Fig. 1 and Fig. 2, and details are not described herein.
It is deposited thereon based on above-mentioned method as depicted in figs. 1 and 2 correspondingly, the present embodiment additionally provides a kind of storage medium Computer program is contained, which realizes the side of synchronization of above-mentioned change data as depicted in figs. 1 and 2 when being executed by processor Method.
Based on this understanding, the technical solution of the application can be embodied in the form of software products, which produces Product can store in a non-volatile memory medium (can be CD-ROM, USB flash disk, mobile hard disk etc.), including some instructions With so that computer equipment (can be personal computer, server or the network equipment an etc.) execution the application is each Method described in implement scene.
Based on above-mentioned method as shown in Figure 1 and Figure 2 and Fig. 3, virtual bench embodiment shown in Fig. 4, in order to realize Above-mentioned purpose, the embodiment of the present application also provides a kind of computer equipments, are specifically as follows personal computer, server, network Equipment etc., the entity device include storage medium and processor;Storage medium, for storing computer program;Processor is used for Computer program is executed to realize the synchronous method of above-mentioned change data as depicted in figs. 1 and 2.
Optionally, which can also include user interface, network interface, camera, radio frequency (Radio Frequency, RF) circuit, sensor, voicefrequency circuit, WI-FI module etc..User interface may include display screen (Display), input unit such as keyboard (Keyboard) etc., optional user interface can also connect including USB interface, card reader Mouthful etc..Network interface optionally may include standard wireline interface and wireless interface (such as blue tooth interface, WI-FI interface).
It will be understood by those skilled in the art that the entity device structure of the synchronizing device of change data provided in this embodiment The restriction to the entity device is not constituted, may include more or fewer components, perhaps combines certain components or difference Component layout.
It can also include operating system, network communication module in storage medium.Operating system is that the above-mentioned computer of management is set The program of standby hardware and software resource, supports the operation of message handling program and other softwares and/or program.Network communication mould Block leads to for realizing the communication between each component in storage medium inside, and between other hardware and softwares in the entity device Letter.
Through the above description of the embodiments, those skilled in the art can be understood that the application can borrow It helps software that the mode of necessary general hardware platform is added to realize, hardware realization can also be passed through.Pass through the skill of application the application Art scheme, compared with currently available technology, the present invention is by the table structure in timing acquiring business party database, while by business The table structure that there is change in party database is compared with the table structure in target database, and will compare inconsistent business The list of modification structure mapping of side realizes that target database is automatically kept with table structure in business party database into target database It is synchronous, reduce a large amount of artificial communication and therefore in the synchronizing process of change data, avoids as caused by artificially linking up Target database and the inconsistent problem of business party database improve the synchronous efficiency of change data.
It will be appreciated by those skilled in the art that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, module in attached drawing or Process is not necessarily implemented necessary to the application.It will be appreciated by those skilled in the art that the mould in device in implement scene Block can according to implement scene describe be distributed in the device of implement scene, can also carry out corresponding change be located at be different from In one or more devices of this implement scene.The module of above-mentioned implement scene can be merged into a module, can also be into one Step splits into multiple submodule.
Above-mentioned the application serial number is for illustration only, does not represent the superiority and inferiority of implement scene.Disclosed above is only the application Several specific implementation scenes, still, the application is not limited to this, and the changes that any person skilled in the art can think of is all The protection scope of the application should be fallen into.

Claims (10)

1. a kind of synchronous method for changing data, which is characterized in that the described method includes:
The table structure that there is variation in unit synchronization time in each business party database is obtained, the change of each business side is obtained Table structure;
The change table structure of each business side is compared with the table structure of target database, it is inconsistent to filter out comparison Each business side change table structure;
The change table structure for comparing inconsistent each business side is parsed, and compares inconsistent each business side for described List of modification structure mapping into the table structure of target database, generate the change table structure of the target database;
Data in the change table structure for comparing inconsistent each business side are extracted as changing data, according to setting in advance The change data are synchronized in the change table structure in target database by the isochronous schedules parameter set.
2. the method according to claim 1, wherein each business number formulary evidence in the acquisition unit synchronization time There is the table structure of variation in library, the change table structure for obtaining each business side includes:
Table structure in timing acquiring unit synchronization time in each business party database, obtains the table structure of each business side;
Starting inspection scheduler task, is detected in the change in the table structure of each business side according to the inspection scheduler task Hold, obtains the change table structure of each business side.
3. the method according to claim 1, wherein the change table structure and mesh by each business side The table structure of mark database is compared, and the change table structure for filtering out the inconsistent each business side of comparison includes:
In the table structure for extracting the metadata and the target database in the change table structure of each business side respectively Metadata;
By the metadata in the table structure of metadata and the target database in the change table structure of each business side It is compared, filters out the change table structure for comparing inconsistent each business side.
4. according to the method described in claim 3, it is characterized in that, described will be in the change table structure of each business side Metadata is compared with the metadata in the table structure of the target database, filters out and compares inconsistent each business side Change table structure include:
By the table structure of metadata corresponding logical construction and the target database in the change table structure of the business side In the corresponding logical construction of metadata be compared, filter out the list of modification knot of the inconsistent each business side of logical construction Structure;
By the table structure of metadata corresponding storage content and the target database in the change table structure of the business side In the corresponding storage content of metadata be compared, filter out the list of modification knot of the inconsistent each business side of storage content Structure.
5. the method according to claim 1, wherein the parsing is described to compare inconsistent each business side Change table structure, and by the table structure of the list of modification structure mapping for comparing inconsistent each business side to target database In, the change table structure for generating the target database includes:
According to the change table structure for comparing inconsistent each business side, it is fixed to generate the corresponding data of the change table structure Adopted sentence;
It is by executing the corresponding data definition statement of the change table structure in the target database, the comparison is different The list of modification structure mapping of each business side caused generates the change of the target database into the table structure of target database Table structure.
6. method according to any one of claims 1-5, which is characterized in that described according to pre-set same step Before the change data are synchronized in the change table structure in target database by degree parameter, which comprises
According to the size of the change data corresponding number, the data volume rank of the change data is determined;
According to the data volume rank of the change data, the corresponding isochronous schedules parameter of the change data is set.
7. according to method described in right 6, which is characterized in that the isochronous schedules parameter include synchronization time parameter, same to step Spend time level parameters and synchronization node weight parameter, the data volume rank according to the change data, described in setting Changing the corresponding isochronous schedules parameter of data includes:
Time required for the change data complete synchronous task, setting are estimated according to the data volume rank of the change data Parameter synchronization time of the change data;
The unit time that the change data are dispatched in synchronous task is estimated according to the data volume rank of the change data, if Set the isochronous schedules time level parameters of the change data;
The change data are estimated according to the data volume rank of the change data to need to open synchronization node in synchronous task The synchronization node weight parameter of the change data is arranged in quantity.
8. a kind of synchronizing device for changing data, which is characterized in that described device includes:
Acquiring unit obtains each for obtaining the table structure that there is variation in unit synchronization time in each business party database The change table structure of a business side;
Screening unit is sieved for the change table structure of each business side to be compared with the table structure of target database Select the change table structure for comparing inconsistent each business side;
Generation unit, for parsing the change table structure for comparing inconsistent each business side, and the comparison is different The list of modification structure mapping of each business side caused generates the change of the target database into the table structure of target database Table structure;
Synchronization unit, for extracting the data in the change table structure for comparing inconsistent each business side as change number According to the change data are synchronized in the change table structure in target database according to pre-set isochronous schedules parameter.
9. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists In the step of processor realizes any one of claims 1 to 7 the method when executing the computer program.
10. a kind of computer storage medium, is stored thereon with computer program, which is characterized in that the computer program is located The step of reason device realizes method described in any one of claims 1 to 7 when executing.
CN201910337189.6A 2019-04-25 2019-04-25 Change synchronous method, device, computer equipment and the computer storage medium of data Pending CN110209730A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910337189.6A CN110209730A (en) 2019-04-25 2019-04-25 Change synchronous method, device, computer equipment and the computer storage medium of data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910337189.6A CN110209730A (en) 2019-04-25 2019-04-25 Change synchronous method, device, computer equipment and the computer storage medium of data

Publications (1)

Publication Number Publication Date
CN110209730A true CN110209730A (en) 2019-09-06

Family

ID=67786356

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910337189.6A Pending CN110209730A (en) 2019-04-25 2019-04-25 Change synchronous method, device, computer equipment and the computer storage medium of data

Country Status (1)

Country Link
CN (1) CN110209730A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110704442A (en) * 2019-09-29 2020-01-17 深圳无域科技技术有限公司 Real-time acquisition method and device for big data
CN111125101A (en) * 2019-12-16 2020-05-08 杭州涂鸦信息技术有限公司 Data center table structure consistency monitoring method and system
CN111431984A (en) * 2020-03-02 2020-07-17 畅捷通信息技术股份有限公司 Application data synchronization method, synchronization device and computer readable storage medium
CN112905706A (en) * 2021-03-19 2021-06-04 平安消费金融有限公司 Database synchronization method and device, storage medium and computer equipment
CN115713262A (en) * 2022-11-21 2023-02-24 河南飙风信息科技有限公司 Data quality inspection platform
CN113672639B (en) * 2021-08-27 2024-04-26 成都长城开发科技股份有限公司 Multi-type database table structure comparison method, system, equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8554762B1 (en) * 2010-12-28 2013-10-08 Amazon Technologies, Inc. Data replication framework
CN104866580A (en) * 2015-05-26 2015-08-26 梁猛 Method for quickly detecting impact caused by database modification to current service
CN107515933A (en) * 2017-08-29 2017-12-26 平安科技(深圳)有限公司 Change method, server, system and the storage medium of source data database table structure
WO2018093094A1 (en) * 2016-11-16 2018-05-24 주식회사 실크로드소프트 Data replacation technique in database management system
US20180285097A1 (en) * 2017-03-28 2018-10-04 Sap Se Zero downtime for frequent software updates
CN109582736A (en) * 2018-11-22 2019-04-05 平安科技(深圳)有限公司 Synchronous method, device and the computer equipment of loan transaction list table

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8554762B1 (en) * 2010-12-28 2013-10-08 Amazon Technologies, Inc. Data replication framework
CN104866580A (en) * 2015-05-26 2015-08-26 梁猛 Method for quickly detecting impact caused by database modification to current service
WO2018093094A1 (en) * 2016-11-16 2018-05-24 주식회사 실크로드소프트 Data replacation technique in database management system
US20180285097A1 (en) * 2017-03-28 2018-10-04 Sap Se Zero downtime for frequent software updates
CN107515933A (en) * 2017-08-29 2017-12-26 平安科技(深圳)有限公司 Change method, server, system and the storage medium of source data database table structure
CN109582736A (en) * 2018-11-22 2019-04-05 平安科技(深圳)有限公司 Synchronous method, device and the computer equipment of loan transaction list table

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
密红;何利力;杨秀梅;: "烟草数据中心ETL技术应用", 计算机系统应用, no. 05 *
郑煜;高宝;: "基于同态哈希的目录服务数据变化捕获方法", 计算机应用研究, no. 07 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110704442A (en) * 2019-09-29 2020-01-17 深圳无域科技技术有限公司 Real-time acquisition method and device for big data
CN111125101A (en) * 2019-12-16 2020-05-08 杭州涂鸦信息技术有限公司 Data center table structure consistency monitoring method and system
CN111125101B (en) * 2019-12-16 2023-10-13 杭州涂鸦信息技术有限公司 Data center table structure consistency monitoring method and system
CN111431984A (en) * 2020-03-02 2020-07-17 畅捷通信息技术股份有限公司 Application data synchronization method, synchronization device and computer readable storage medium
CN111431984B (en) * 2020-03-02 2023-04-07 畅捷通信息技术股份有限公司 Application data synchronization method, synchronization device and computer readable storage medium
CN112905706A (en) * 2021-03-19 2021-06-04 平安消费金融有限公司 Database synchronization method and device, storage medium and computer equipment
CN113672639B (en) * 2021-08-27 2024-04-26 成都长城开发科技股份有限公司 Multi-type database table structure comparison method, system, equipment and storage medium
CN115713262A (en) * 2022-11-21 2023-02-24 河南飙风信息科技有限公司 Data quality inspection platform

Similar Documents

Publication Publication Date Title
CN110209730A (en) Change synchronous method, device, computer equipment and the computer storage medium of data
CN106776894B (en) Log database system and synchronous method
CN105608203B (en) A kind of Internet of Things log processing method and device based on Hadoop platform
CN100375047C (en) Management method for computer log
TW202002587A (en) Block chain-based data processing method and device
CN104317843B (en) A kind of data syn-chronization ETL system
CN108536752A (en) A kind of method of data synchronization, device and equipment
CN106933836B (en) Data storage method and system based on sub-tables
CN105718565A (en) Data warehouse model construction method and construction apparatus
CN109086410A (en) The processing method and system of streaming mass data
CN109634970A (en) Table method of data synchronization, equipment, storage medium and device
CN110321383A (en) Big data platform method of data synchronization, device, computer equipment and storage medium
CN109831486A (en) The background data server system and data processing method of multi-client
US10891129B1 (en) Decentralized development operations blockchain system
CN110062029A (en) Change data synchronous method, apparatus, computer equipment and computer storage medium
CN111475490A (en) Data management system and method for data directory system
US10963963B2 (en) Rule based hierarchical configuration
CN108446326A (en) A kind of isomeric data management method and system based on container
CN110442651A (en) A method of it is uploaded automatically based on kettle realization excel data and triggers scheduling
van Dongen et al. EMiT: A process mining tool
CN116662441A (en) Distributed data blood margin construction and display method
CN109298929A (en) Timing task carrying-out time recommended method, device, equipment and storage medium
CN112527801A (en) Data synchronization method and system between relational database and big data system
CN115640300A (en) Big data management method, system, electronic equipment and storage medium
Bellavista et al. Design guidelines for big data gathering in industry 4.0 environments

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination