CN110222117A - A kind of data conversion synchronous method, equipment and the storage medium of heterogeneous database - Google Patents
A kind of data conversion synchronous method, equipment and the storage medium of heterogeneous database Download PDFInfo
- Publication number
- CN110222117A CN110222117A CN201910435627.2A CN201910435627A CN110222117A CN 110222117 A CN110222117 A CN 110222117A CN 201910435627 A CN201910435627 A CN 201910435627A CN 110222117 A CN110222117 A CN 110222117A
- Authority
- CN
- China
- Prior art keywords
- column
- operation table
- data conversion
- carry out
- transfer function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention discloses data conversion synchronous method, equipment and the storage mediums of a kind of heterogeneous database, the described method includes: creating transfer function first, then it reads configuration file and loads out the table name and column information for needing to carry out data processing, and verify the validity of configuration；Later when verifying effective, receives the log information packet that source is sent and parse；Judge whether the operation table in log information packet needs to carry out data conversion according to the need table name for carrying out data processing and column information that load out later；Finally when needing to carry out data conversion, handled according to the replacement that transfer function carries out expression formula to the operation table, and be applied in target database.The present invention effectively meets that real-time data are synchronous and conversion process, and is calculated using built-in function and change data, high efficiency, and has the scalability of stronger flexibility and height.
Description
Technical field
The present invention relates to database synchronization technical fields, and in particular to a kind of data conversion side of synchronization of heterogeneous database
Method, equipment and storage medium.
Background technique
With the continuous development of business event scale, the load pressure that the information system of enterprise's application is faced is served
Also increasing, while the diversified type of service of enterprise causes the demand accessed data of information system increasingly sophisticated and various
Change.In the epoch of information technology, Modern Information system has become the key of corporate operation, stores the core data money of enterprise
It produces.How storage and management effectively to be carried out to data, construct efficiently available information system, guarantee the continuity of business,
Meet diversified business demand, ensure business data assets to greatest extent, High-effective Service user is current enterprise informationization
Required facing challenges in systems development process.
For the safety for ensureing data storage, for Modern Operations System, it usually needs use related disaster recovery backup technology structure
Build disaster recovery and backup systems.The traditional backup/restoration system based on database itself, server cluster system, disk array, Oracle
The technologies such as DataGuard are usually constructed with certain usage scenario and limitation, can not fully meet enterprise for information system not in
The requirement of disconnected service.For traditional database standby system, the data meeting if server goes wrong between data backup twice
It loses, and in the biggish situation of data volume, need to usually spend longer time using backup and reduction；Server cluster system
It is unable to reach remote backup purpose, and expensive；Disk array is unable to cope with delay machine caused by operating system or application program,
And service can not be recovered immediately；The products such as Oracle DataGuard are centralized database solutions, and standby host must be with master
Machine isomorphism, using identical database version, entire standby host database can only be accessed with read-only mode, and it is whole also to cannot achieve data
Conjunction and data distribution.
In addition, the diversified business demand of enterprise results in the need for carrying out diversified processing to data in big data era,
It is every using needs to meet.Data pick-up that traditional resource information integration is provided generally by means of specific purpose tools such as ETL, turn
The functions such as change and clean.ETL data processing tools create triggering on realization mechanism it is generally necessary in Production database system
Device, shadow table etc. realize that incremental data extracts.In complex business systems and mass data processing, this can be to production system
It can be produced bigger effect with stability.Another aspect ETL tool is realized at data conversion often through planned task schedule
Reason, is not able to satisfy the demand of real-time data synchronization conversion process.
Thus the prior art needs to improve.
Summary of the invention
Place in view of above-mentioned deficiencies of the prior art, the purpose of the present invention is to provide a kind of data of heterogeneous database to turn
Synchronous method, equipment and storage medium are changed, to solve the problems, such as that current data can not real-time synchronization and efficient conversion processing.
In order to achieve the above object, this invention takes following technical schemes:
A kind of data conversion synchronous method of heterogeneous database, includes the following steps:
Create transfer function, wherein the transfer function is used to define the expression formula for needing the column converted and replacing the column；
It reads configuration file and loads out the table name and column information for needing to carry out data processing, and verify the effective of configuration
Property；
When verifying configuration is effective, receives the log information packet that source is sent and parse；
The table name for carrying out data processing and column information is needed to judge the operation table in log information packet according to what is loaded out
Whether need to carry out data conversion；
When judging that the operation table in log information packet needs to carry out data conversion, according to transfer function to the operation table into
The replacement of row expression is handled, and operation table is applied in target database by treated.
Preferably, in the data conversion synchronous method of the heterogeneous database, the log information packet includes at least thing
Business ID, transaction operation type and operation table, the operation table include at least table name, column information and data value.
Preferably, in the data conversion synchronous method of the heterogeneous database, the creation transfer function the step of
Later, it is reading configuration file and is loading out the table column name information for needing to carry out data processing and external data processing dynamically
Library, and include: before the step of verifying the validity of external data processing dynamic base
Using configuration file to the column for needing the table name for carrying out data processing, needs to convert and the expression formula for replacing the column
It is configured.
Preferably, described to need to carry out according to what is loaded out in the data conversion synchronous method of the heterogeneous database
The table name and column information of data processing judge whether the operation table in log information packet needs the step of carrying out data conversion packet
It includes:
The operation table in log information packet is extracted, the table name of the operation table needs to carry out at data with what is loaded out
The table name of reason；
When the table name of the operation table is consistent with the table name for needing to carry out data processing loaded out, by the operation table
Column information be compared with the column information for carrying out data processing is needed；
Judged to whether there is column to be converted in the column information in operation table according to comparison result, if comparison result one
It causes, then judges that there are column to be converted in the operation table, otherwise judging the operation table, there is no column to be converted.
Preferably, described to judge in log information packet in the data conversion synchronous method of the heterogeneous database
It when operation table needs to carry out data conversion, is handled according to the replacement that transfer function carries out expression formula to the operation table, and will processing
Operation table afterwards is applied to the step in target database
When judging to call the transfer function of the column there are when column to be converted in operation table；
According to the transfer function of the column column are carried out with the replacement of expression formula；
Replaced column information is assembled into corresponding SQL statement and is applied in target database.
Preferably, in the data conversion synchronous method of the heterogeneous database, the transfer function pair according to the column
The step of replacement of column progress expression formula, specifically includes:
The expression formula of the column is replaced according to the transfer function that the conversion arranges, and is existed in the parameter of the transfer function of the column
When to the references of other column, replicates the train value of referenced column and fill it into the expression formula of replacement.
Preferably, the data conversion synchronous method of the heterogeneous database further include:
When the table name for judging the operation table in log information packet does not need to carry out to be not present in data conversion or operation table
When column to be converted, the column information in the operation table is assembled into corresponding SQL statement and is applied to target database.
Preferably, in the data conversion synchronous method of the heterogeneous database, source hair is received by TCP/IP network
The log information packet sent.
A kind of data conversion synchronizer of heterogeneous database comprising: processor, memory and communication bus；
The computer-readable program that can be executed by the processor is stored on the memory；
The communication bus realizes the connection communication between processor and memory；
The processor realizes the data conversion of heterogeneous database as described above when executing the computer-readable program
Step in synchronous method.
A kind of computer readable storage medium, the computer-readable recording medium storage have one or more program,
One or more of programs can be executed by one or more processor, to realize the number of heterogeneous database as described above
According to the step in conversion synchronization method.
In the data conversion synchronous method of heterogeneous database provided by the invention, equipment and storage medium, the method packet
Include: then creation transfer function first reads configuration file and loads out the table name and column information for needing to carry out data processing,
And verify the validity of configuration；Later when verifying configuration is effective, receives the log information packet that source is sent and parse；Root later
According to load out need the table name for carrying out data processing and column information judge the operation table in log information packet whether need into
Row data conversion；Finally when judging that the operation table in log information packet needs to carry out data conversion, according to transfer function to this
The replacement processing that table carries out expression formula is operated, and operation table is applied in target database by treated.One aspect of the present invention
Based on the real time data synchronization of database journal parsing, independent of the trigger or the technological means such as shadow table on source host,
Incremental data library data are obtained, the influence to source Production database system is smaller；On the other hand, the present invention effectively meets reality
When data are synchronous and conversion process, and calculated using built-in function and change data, not only high efficiency, and have
The scalability of standby stronger flexibility and height, the function that can enrich constantly realize a variety of demands to the function of data processing
Under data conversion treatment.
Detailed description of the invention
Fig. 1 is the flow chart of the data conversion synchronous method of heterogeneous database provided by the invention.
Fig. 2 is the flow chart of the step S400 in the data conversion synchronous method of heterogeneous database provided by the invention.
Fig. 3 is the flow chart of the step S500 in the data conversion synchronous method of heterogeneous database provided by the invention.
Fig. 4 is the flow chart of a preferred embodiment of the data conversion synchronous method of heterogeneous database provided by the invention.
Fig. 5 is the running environment schematic diagram of the preferred embodiment of the data conversion synchronization program of heterogeneous database of the present invention.
Fig. 6 is the functional module of the system preferred embodiment of the data conversion synchronization program of present invention installation heterogeneous database
Figure.
Specific embodiment
The disadvantages of there are limitations in view of the solution of database data disaster tolerance and conversion process in the prior art, it is of the invention
It is designed to provide data conversion synchronous method, equipment and the storage medium of a kind of heterogeneous database, it can be to heterogeneous database
Data carry out implementing synchronous and efficient conversion processing.
To make the purpose of the present invention, technical solution and effect clearer, clear and definite, right as follows in conjunction with drawings and embodiments
The present invention is further described.It should be appreciated that described herein, specific examples are only used to explain the present invention, is not used to
Limit the present invention.
Referring to Fig. 1, the data conversion synchronous method of heterogeneous database provided by the invention, includes the following steps:
S100, creation transfer function, wherein the transfer function is used to define the table for needing the column converted and replacing the column
Up to formula.
Specifically, the present invention defined by custom function the column for needing to need to replace in the operation table replaced and
The expression formula for replacing the column, when needing to carry out data conversion, the result of data conversion is automatically generated by replacement function, such as will
The sum of C1 and C2 column are converted to when C1 is listed in synchronous, since the transfer function in the present invention can be customized by the user, therefore this hair
It is bright be directed to different demands when, data conversion treatment can be carried out using different replacement functions, and in follow-up developments
Can also by increasing the quantity of transfer function come the function for data processing of enriching constantly, with stronger flexibility and height can
Scalability, it should be noted that the transfer function in the present invention can be one or more, each transfer function defines one
The column that need to convert and the expression formula for replacing the column avoid that showing for data conversion disorder occurs during data conversion treatment
As in a specific embodiment, a specific transfer function is as follows:
Above-mentioned function realizes the function that is added of incoming parameter C1 with C2 value, and returns to the result after being added.
S200, it reads configuration file and loads out the table name and column information for needing to carry out data processing, and verify configuration
Validity.
Specifically, the configuration file is the configuration file of XML format, for the table name for needing to carry out data processing
And column name is configured, in a specific embodiment, the configuration file of the XML format is as follows:
In other words, in the present embodiment, using table functional label configuration data conversion process function, wherein
The affiliated mode of table of schema tag representation conversion；Name label is then the specified table name for wanting conversion table；Col functional label is used for
The column information of configuration data conversion and the expression formula for replacing the column.When target side synchronous service starts, XML format is read first
Configuration file, load needs to carry out the table information of data processing, and carries out validation verification to configuration, configures testing for validity
Card, which refers to, can find the column information for needing to convert under the table name by table name, and the replacement of the column can be found by the column information
Expression formula, if may be implemented in verifying, indicating to be verified, otherwise indicate authentication failed.The present invention is in order to guarantee number
It needs to verify the validity of configuration before carrying out data conversion according to the success rate of conversion, when verifying effective, can continue
Carry out data conversion.
Preferably, after the step S100, before the step S200 further include:
Using configuration file to the column for needing the table name for carrying out data processing, needs to convert and the expression formula for replacing the column
It is configured.That is, before carrying out data conversion, need replacement expression formula when converting each table and the column information into
Row configuration can directly recall the replacement expression formula of the column in conversion by the column information, have stronger flexibility.
S300, verifying configuration it is effective when, receive source send log information packet simultaneously parse.
Specifically, the table for needing to carry out data conversion is sent the past by source in the form of log information packet, therefore source
The generation for also needing to carry out log information packet before carrying out the transmission of log information packet, due in heterogeneous database system environment
In, Incremental Log information caused by source database can not directly apply to target client database, therefore need by source
The log information of client database is parsed, and the message package of specific format inside synchronous service is then generated, and is stored in and is sent team
Log sending module is waited to be sent in column, when it is implemented, the log trapping module of source data synchronization service is based on day
Will capture technique captures the Incremental Log data of source Database Systems in real time, Incremental Log data is then passed to log
Analysis module is parsed, later the log analyzing module of source data synchronization service, Incremental Log is resolved to internal specific
Message packet format is stored in transmit queue.Preferably, the log information packet includes at least affairs ID, transaction operation type
And operation table, the operation table include at least table name, column information and data value.Target side carry out data it is synchronous when, connecing
The operation received is classified according to affairs ID.Later, the log sending module of source data synchronization service is based on thread synchronization machine
System, the log information packet in internal transmit queue is sent in target side data synchronization service system by TCP/IP network,
In other words, the present invention receives the log information packet that source is sent by TCP/IP network.Preferably, in order to guarantee data transmit
Reliability and data integrity, source log sending module takes complete acknowledgement messaging mechanism, and log sending module is only
Have and just think that data are transmitted after obtaining target side confirmation message, otherwise will transmit data again automatically, target side is connecing
After receiving log information packet, sequence number is carried out to each log information packet while needing to return to confirmation message.In addition, in source
While sending log information packet, target side also needs to verify the validity of external data conversion dynamic base, when effective, can just connect
The log information packet that source is sent is received, does not otherwise receive the log information packet of source transmission；The present invention can be independent of source master
The technological means such as trigger or shadow table on machine, to obtain incremental data library data, to the shadow of source Production database system
Sound is smaller.
Furthermore, it is understood that the present invention can also carry out log information packet after the log information packet for receiving source transmission
Parsing, affairs ID, transaction operation type and the operation table information extracted in log information packet will then as unit of affairs
Log information packet is distributed to corresponding log execution thread and is executed, in addition, it should be noted that, the present invention is receiving one
Next log information packet is immediately subjected to after a log information packet, that is to say, that the present invention can carry out simultaneously multiple data
Conversion, the present invention are configured with multiple log execution threads, are respectively intended to execute different log information packets, each log executes line
Cheng Binghang is executed, and can effectively improve the efficiency of data conversion.
S400, the table name for carrying out data processing and column information is needed to judge the behaviour in log information packet according to what is loaded out
Make whether table needs to carry out data conversion.
Specifically, the present invention is after extracting the table name and column information in log information packet, by with need to carry out
The table name and column information of data processing conversion judge whether to need to carry out data conversion again after being compared, and avoid invalid day
Will conversion process, specific deterministic process please refer to Fig. 2.
Referring to Fig. 2, it is the flow chart of the step S400, include the following steps:
Operation table in S401, extraction log information packet, the table name of the operation table need to carry out with what is loaded out
The table name of data processing；
S402, when the table name of the operation table is consistent with the table name for needing to carry out data processing loaded out, will be described
The column information of operation table is compared with the column information for carrying out data processing is needed；
S403, judge to operate with the presence or absence of column to be converted in the column information in table, if comparing knot according to comparison result
Fruit is consistent, then judges that there are column to be converted in the operation table, and otherwise judging the operation table, there is no column to be converted.
In other words, the present invention is carrying out when whether data conversion treatment being needed to judge of operation table, needs by two
Step, it is necessary first to compare table name information, judge whether the table name of the operation table is the table for needing to carry out data conversion, if not
It is not have to the column information for verifying the operation table directly then, in-stockroom operation can be directly carried out, if it is, needing to verify this again
The column information of table is operated, when it is implemented, extracting the column information of the operation table, each operation table includes multiple column informations,
Each column information of the operation table is compared with the column information for carrying out data conversion is needed, judges that each column information is one by one
No is the column for needing to carry out data conversion, if the column name of the column is the column name for needing to carry out data conversion of configuration, then it represents that
The column need to carry out data conversion, and otherwise the table name column do not need to carry out data conversion, can directly execute in-stockroom operation.
S500, when judging that the operation table in log information packet needs to carry out data conversion, according to transfer function to the behaviour
Make the replacement processing that table carries out expression formula, and operation table is applied in target database by treated.
Specifically, the present invention carries out data conversion to operation table by way of calling directly transfer function, specifically
When conversion, processing is ignored to the column for being not necessary to carry out data conversion in operation table, the column for needing to carry out data conversion of table will be operated
Independent data processing conversion is carried out, specific data conversion process please refers to Fig. 3.
Referring to Fig. 3, it is the flow chart of the step S500, include the following steps:
S501, when judging to call the transfer function of the column there are when column to be converted in operation table；
S502, the replacement that according to the transfer function of the column column are carried out with expression formula；
S503, replaced column information is assembled into corresponding SQL statement and is applied in target database.
Specifically, the affairs are transferred to after target side synchronous service receives the operation of the submission comprising operation table (T table)
Execution part is put in storage, and is assembled into SQL statement specifically to carry out in-stockroom operation, the type of SQL statement generally comprises
INSERT sentence, UPDATE statement and DELETE sentence, therefore in-stockroom operation includes INSERT operation, UPDATE operation and DELETE
Operation, for example, the storage SQL of INSERT operation will be converted into INSERT INTO T (C1, C2) VALUES (FUN_C1_
ADD_C2 (: 1: 2): 3)；Function expression that C1 train value part will be configured is replaced；The storage SQL of UPDATE operation
Will be converted into UPDATE T SET C1=FUN_C1_ADD_C2 (: 1: 2), C2=:3 WHERE C1=FUN_C1_
ADD_C2 (: 4: 5) AND C2=C6, it can be seen that, the old value in the new value and WHERE condition of C1 column is all by function from SQL
Expression formula replacement；The storage SQL of DELETE operation will be converted into DELETE FROM T WHERE C1=FUN_C1_ADD_
C2 (: 1: 2) value of the C1 column in AND C2=:3, SQL is replaced by function expression.
Further, referring to Fig. 4, the step S503 is specifically included:
The expression formula of the column is replaced according to the transfer function that the conversion arranges, and is existed in the parameter of the transfer function of the column
When to the references of other column, replicates the train value of referenced column and fill it into the expression formula of replacement.
Specifically, since when progress transfer function defines, the conversion of certain data types may use remaining column
Train value, if undoubtedly will increase the complexity of function at this point, write these specific train values exactly directly in transfer function,
And may also need multiple functions to be defined, therefore the present invention in the parameter to transfer function exist other column are drawn
With the case where, also duplication referenced column train value and be filled in expression formula, with reach data conversion as a result, and avoiding
User needs to define complicated transfer function, reduces the load of system.
Preferably, the data conversion synchronous method of heterogeneous database provided in this embodiment further include:
When the table name for judging the operation table in log information packet does not need to carry out to be not present in data conversion or operation table
When column to be converted, the column information in the operation table is assembled into corresponding SQL statement and is applied to target database.
In order to better understand the present invention, it elaborates below in conjunction with Fig. 5 to technical solution of the present invention:
Then step 1, the Incremental Log data for capturing source Database Systems in real time by source log trapping module will increase
Amount daily record data passes to log analysis module.
Incremental Log is resolved to internal particular message packet format by source log analyzing module by step 2, is stored in transmission
In queue.
Step 3 is sent out the log information packet in internal transmit queue by TCP/IP network by source log sending module
Give target side data synchronization service system.
Step 4, target side data synchronous system are using configuration file to needing the table name for carrying out data processing, need to convert
Column and replace the expression formulas of the column and configured.
Step 5, target side data synchronous system read configuration file and load out the table name that needs to carry out data processing with
And column information, and the validity of configuration is verified, step 6 is executed when verifying effective, otherwise repeatedly step 5.
Step 6, the log information packet for receiving source transmission simultaneously parse.
Step 7 needs the table name for carrying out data processing and column information to judge in log information packet according to what is loaded out
Whether the table name of operation table is the table name for needing to carry out data conversion, no to then follow the steps 10 if it is thening follow the steps 8.
Whether step 8, the column information for judging the operation table are the column informations for needing to carry out data processing, are if it is held
Row step 9, it is no to then follow the steps 10.
Step 9, the transfer function for calling the column, and according to the transfer function of the column to the column carry out expression formula replacement after
It is assembled into corresponding SQL statement and is applied in target database.
Column information in log information packet is assembled into corresponding SQL statement and is applied in target database by step 10.
The real time data synchronization that one aspect of the present invention is parsed based on database journal, independent of the trigger on source host
Or the technological means such as shadow table, to obtain incremental data library data, the influence to source Production database system is smaller；Another party
Face, the present invention effectively meet that real-time data are synchronous and conversion process, and are calculated using built-in function and change data,
Not only high efficiency, and have the scalability of stronger flexibility and height, the function that can enrich constantly is to data processing
Function, realize the data conversion treatment under a variety of demands.
As shown in figure 5, the data conversion synchronous method based on above-mentioned heterogeneous database, the present invention further correspondingly provide one kind
The data conversion synchronizer of heterogeneous database, the data conversion synchronizer of the heterogeneous database can be mobile terminal,
Desktop PC, notebook, palm PC and server etc. calculate equipment.The data conversion synchronizer of the heterogeneous database
Including processor 10, memory 20 and display 30.Fig. 5 illustrates only the part of the data conversion synchronizer of heterogeneous database
Component, it should be understood that it is not required for implementing all components shown, more or less groups of the implementation that can be substituted
Part.
The memory 20 can be the interior of the data conversion synchronizer of the heterogeneous database in some embodiments
Portion's storage unit, for example, heterogeneous database data conversion synchronizer hard disk or memory.The memory 20 is at other
It is also possible to the External memory equipment of the data conversion synchronizer of the heterogeneous database, such as the isomery number in embodiment
According to the plug-in type hard disk being equipped on the data conversion synchronizer in library, intelligent memory card (Smart Media Card, SMC), peace
Digital (Secure Digital, SD) card, flash card (Flash Card) etc..
Further, the memory 20 can also both include that the inside of the data conversion synchronizer of heterogeneous database is deposited
Storage unit also includes External memory equipment.The memory 20 be used for store be installed on the heterogeneous database data conversion it is same
Walk the application software and Various types of data of equipment, such as the program code of the data conversion synchronizer for installing heterogeneous database
Deng.The memory 20 can be also used for temporarily storing the data that has exported or will export.In one embodiment, it deposits
The data conversion synchronization program 40 of heterogeneous database, the data conversion synchronization program 40 of the heterogeneous database are stored on reservoir 20
It can be performed by processor 10, to realize the data conversion synchronous method of the heterogeneous database of each embodiment of the application.
The processor 10 can be in some embodiments a central processing unit (Central Processing Unit,
CPU), microprocessor or other data processing chips, for running the program code stored in the memory 20 or processing number
According to, such as data conversion synchronous method for executing the heterogeneous database etc..
The display 30 can be light-emitting diode display, liquid crystal display, touch-control liquid crystal display in some embodiments
And OLED (Organic Light-Emitting Diode, Organic Light Emitting Diode) touches device etc..The display 30 is used
In the information for the data conversion synchronizer for being shown in the heterogeneous database and for showing visual user interface.Institute
The component 10-30 for stating the data conversion synchronizer of heterogeneous database is in communication with each other by system bus.
In one embodiment, when processor 10 executes the data conversion synchronization program of heterogeneous database in the memory 20
It is performed the steps of when 40
Create transfer function, wherein the transfer function is used to define the expression formula for needing the column converted and replacing the column；
It reads configuration file and loads out the table name and column information for needing to carry out data processing, and verify the effective of configuration
Property；
When verifying configuration is effective, receives the log information packet that source is sent and parse；
The table name for carrying out data processing and column information is needed to judge the operation table in log information packet according to what is loaded out
Whether need to carry out data conversion；
When judging that the operation table in log information packet needs to carry out data conversion, according to transfer function to the operation table into
The replacement of row expression is handled, and operation table is applied in target database by treated.
The log information packet includes at least affairs ID, transaction operation type and operation table, the operation table and at least wraps
Include table name, column information and data value.
Further, it after the creation transfer function the step of, needs to carry out reading configuration file and loading out
The table column name information and external data of data processing handle dynamic base, and verify the validity of external data processing dynamic base
Include: before step
Using configuration file to the column for needing the table name for carrying out data processing, needs to convert and the expression formula for replacing the column
It is configured.
Further, in the data conversion synchronizer of the heterogeneous database, it is described according to the needs loaded out into
The table name and column information of row data processing judge whether the operation table in log information packet needs the step of carrying out data conversion
Include:
The operation table in log information packet is extracted, the table name of the operation table needs to carry out at data with what is loaded out
The table name of reason；
When the table name of the operation table is consistent with the table name for needing to carry out data processing loaded out, by the operation table
Column information be compared with the column information for carrying out data processing is needed；
Judged to whether there is column to be converted in the column information in operation table according to comparison result, if comparison result one
It causes, then judges that there are column to be converted in the operation table, otherwise judging the operation table, there is no column to be converted.
Further, described when judging that the operation table in log information packet needs to carry out data conversion, according to conversion letter
Several replacements for carrying out expression formula to the operation table are handled, and operation table is applied to the step packet in target database by treated
It includes:
When judging to call the transfer function of the column there are when column to be converted in operation table；
According to the transfer function of the column column are carried out with the replacement of expression formula；
Replaced column information is assembled into corresponding SQL statement and is applied in target database.
Further, the step of transfer function according to the column carries out the replacement of expression formula to the column specifically includes:
The expression formula of the column is replaced according to the transfer function that the conversion arranges, and is existed in the parameter of the transfer function of the column
When to the references of other column, replicates the train value of referenced column and fill it into the expression formula of replacement.
Further, it is gone back when processor 10 executes the data conversion synchronization program 40 of heterogeneous database in the memory 20
It realizes:
When the table name for judging the operation table in log information packet does not need to carry out to be not present in data conversion or operation table
When column to be converted, the column information in the operation table is assembled into corresponding SQL statement and is applied to target database.
Preferably, the log information packet that source is sent is received by TCP/IP network.
Referring to Fig. 6, its system preferred embodiment for the data conversion synchronization program of present invention installation heterogeneous database
Functional block diagram.In the present embodiment, the system for installing the data conversion synchronization program of heterogeneous database can be divided into
One or more modules, one or more of modules are stored in the memory 20, and are handled by one or more
Device (the present embodiment is the processor 10) is performed, to complete the present invention.For example, installing the number of heterogeneous database in Fig. 6
Function creation module 21, configuration file read module 22, data reception can be divided into according to the system of conversion synchronization program
Block 23, data conversion judgment module 24 and data conversion module 25.The so-called module of the present invention is to refer to complete specific function
Series of computation machine program instruction section, exist than program more suitable for describing the data conversion synchronization program of the heterogeneous database
Implementation procedure in the data conversion synchronizer of the heterogeneous database.The module 21-25 will specifically be introduced by being described below
Function.
Function creation module 21, for creating transfer function, wherein the transfer function is used to define the column for needing to convert
With the expression formula for replacing the column；
Configuration file read module 22, for read configuration file and load out the table name for needing to carry out data processing and
Column information, and verify the validity of configuration；
Data reception module 23, for receiving the log information packet that source is sent and parsing when verifying configuration is effective；
Data conversion judgment module 24, for needing the table name for carrying out data processing and column information to sentence according to what is loaded out
Whether the operation table in disconnected log information packet needs to carry out data conversion；
Data conversion module 25, for when judging that the operation table in log information packet needs to carry out data conversion, according to
The replacement that transfer function carries out expression formula to the operation table is handled, and operation table is applied in target database by treated.
Wherein, the log information packet includes at least affairs ID, transaction operation type and operation table, and the operation table is extremely
It less include table name, column information and data value.
Further, the system of the data conversion synchronization program of heterogeneous database is installed further include:
Configuration file generation module, for using configuration file to needing the table name that carries out data processing, needs to convert
The expression formula for arranging and replacing the column is configured.
Data conversion judgment module 24 specifically includes:
Table name information comparing unit, for extracting the operation table in log information packet, the table name of the operation table and
What is loaded out needs to carry out the table name of data processing；
Column information comparing unit, for the table name in the operation table and the table name for needing to carry out data processing loaded out
When consistent, the column information of the operation table is compared with the column information for carrying out data processing is needed；
Judging unit whether there is column to be converted in the column information in operation table for judging according to comparison result, such as
Fruit comparison result is consistent, then judges that there are column to be converted in the operation table, otherwise judges that the operation table is not present wait turn
The column changed.
The data conversion module 25 specifically includes:
Function calling cell, for when judging to call the transfer function of the column there are when column to be converted in operation table；
Replacement unit carries out the replacement of expression formula for the transfer function according to the column to the column；
Output unit, for replaced column information to be assembled into corresponding SQL statement and is applied in target database.
The replacement unit is specifically used for:
The expression formula of the column is replaced according to the transfer function that the conversion arranges, and is existed in the parameter of the transfer function of the column
When to the references of other column, replicates the train value of referenced column and fill it into the expression formula of replacement.
Further, the system of the data conversion synchronization program of heterogeneous database is installed further include:
Direct application module, for when the table name that judge operation table in log information packet do not need progress data conversion or
Person operates in table there is no when column to be converted, and the column information in the operation table is assembled into corresponding SQL statement and is applied
To target database.
Preferably, the log information packet that source is sent is received by TCP/IP network.
In conclusion in the data conversion synchronous method of heterogeneous database provided by the invention, equipment and storage medium, institute
The method of stating includes: creation transfer function first, then read configuration file and load out the table name that needs to carry out data processing with
And column information, and verify the validity of configuration；Later when verifying configuration is effective, receives the log information packet that source is sent and solve
Analysis；The table name for carrying out data processing and column information is needed to judge that the operation table in log information packet is according to what is loaded out later
It is no to need to carry out data conversion；Finally when judging that the operation table in log information packet needs to carry out data conversion, according to conversion
The replacement that function carries out expression formula to the operation table is handled, and operation table is applied in target database by treated.This hair
The real time data synchronization that bright one side is parsed based on database journal, independent of the trigger or the skills such as shadow table on source host
Art means, to obtain incremental data library data, the influence to source Production database system is smaller；On the other hand, the present invention has
Effect meets that real-time data are synchronous and conversion process, and is calculated using built-in function and change data, not only high-efficient
Effect, and have the scalability of stronger flexibility and height, the function that can enrich constantly realizes the function of data processing
Data conversion treatment under a variety of demands.
Certainly, those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method,
It is that related hardware (such as processor, controller etc.) can be instructed to complete by computer program, the program can store
In a computer-readable storage medium, which may include the process such as above-mentioned each method embodiment when being executed.Its
Described in storage medium can be for memory, magnetic disk, CD etc..
It should be understood that the application of the present invention is not limited to the above for those of ordinary skills can
With improvement or transformation based on the above description, all these modifications and variations all should belong to the guarantor of appended claims of the present invention
Protect range.
Claims (10)
1. a kind of data conversion synchronous method of heterogeneous database, which comprises the steps of:
Create transfer function, wherein the transfer function is used to define the expression formula for needing the column converted and replacing the column；
It reads configuration file and loads out the table name and column information for needing to carry out data processing, and verify the validity of configuration；
When verifying configuration is effective, receives the log information packet that source is sent and parse；
Need whether the table name for carrying out data processing and column information judge the operation table in log information packet according to what is loaded out
It needs to carry out data conversion；
When judging that the operation table in log information packet needs to carry out data conversion, table is carried out to the operation table according to transfer function
Replacement up to formula is handled, and operation table is applied in target database by treated.
2. the data conversion synchronous method of heterogeneous database according to claim 1, which is characterized in that the log information
Packet includes at least affairs ID, transaction operation type and operation table, the operation table and includes at least table name, column information and data
Value.
3. the data conversion synchronous method of heterogeneous database according to claim 2, which is characterized in that turn in the creation
After the step of exchange the letters number, need to carry out the table column name information and external number of data processing reading configuration file and loading out
Include: according to processing dynamic base, and before the step of verifying the validity of external data processing dynamic base
The column for needing the table name for carrying out data processing, needs to convert and the expression formula for replacing the column are carried out using configuration file
Configuration.
4. the data conversion synchronous method of heterogeneous database according to claim 3, which is characterized in that described according to load
The need table name for carrying out data processing and column information out judges whether the operation table in log information packet needs to carry out data
The step of conversion includes:
Extract the operation table in log information packet, the table name of the operation table and what is loaded out need to carry out data processing
Table name；
When the table name of the operation table is consistent with the table name for needing to carry out data processing loaded out, by the column of the operation table
Information is compared with the column information for carrying out data processing is needed；
Judged to whether there is column to be converted in the column information in operation table according to comparison result, if comparison result is consistent,
Judge that there are column to be converted in the operation table, otherwise judging the operation table, there is no column to be converted.
5. the data conversion synchronous method of heterogeneous database according to claim 4, which is characterized in that described to judge day
When operation table in will message package needs to carry out data conversion, according to transfer function to the operation table carry out expression formula replacement at
Reason, and operation table is applied to the step in target database includes: by treated
When judging to call the transfer function of the column there are when column to be converted in operation table；
According to the transfer function of the column column are carried out with the replacement of expression formula；
Replaced column information is assembled into corresponding SQL statement and is applied in target database.
6. the data conversion synchronous method of heterogeneous database according to claim 5, which is characterized in that described according to the column
Transfer function to the column carry out expression formula replacement the step of specifically include:
The expression formula of the column is replaced according to the transfer function that the conversion arranges, and is existed in the parameter of the transfer function of the column to it
When the reference that it is arranged, replicates the train value of referenced column and fill it into the expression formula of replacement.
7. the data conversion synchronous method of heterogeneous database according to claim 6, which is characterized in that further include:
When the table name for judging the operation table in log information packet does not need to carry out to be not present in data conversion or operation table wait turn
When the column changed, the column information in the operation table is assembled into corresponding SQL statement and is applied to target database.
8. the data conversion synchronous method of heterogeneous database described in -7 any one according to claim 1, which is characterized in that logical
It crosses TCP/IP network and receives the log information packet that source is sent.
9. a kind of data conversion synchronizer of heterogeneous database characterized by comprising processor, memory and communication are total
Line；
The computer-readable program that can be executed by the processor is stored on the memory；
The communication bus realizes the connection communication between processor and memory；
The processor realizes the isomeric data as described in claim 1-8 any one when executing the computer-readable program
Step in the data conversion synchronous method in library.
10. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage have one or
Multiple programs, one or more of programs can be executed by one or more processor, to realize that claim 1-8 such as appoints
Step in the data conversion synchronous method of heterogeneous database described in meaning one.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910435627.2A CN110222117A (en) | 2019-05-23 | 2019-05-23 | A kind of data conversion synchronous method, equipment and the storage medium of heterogeneous database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910435627.2A CN110222117A (en) | 2019-05-23 | 2019-05-23 | A kind of data conversion synchronous method, equipment and the storage medium of heterogeneous database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110222117A true CN110222117A (en) | 2019-09-10 |
Family
ID=67817866
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910435627.2A Pending CN110222117A (en) | 2019-05-23 | 2019-05-23 | A kind of data conversion synchronous method, equipment and the storage medium of heterogeneous database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110222117A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110659263A (en) * | 2019-09-24 | 2020-01-07 | 浪潮软件集团有限公司 | Heterogeneous database data exchange method |
| CN110727646A (en) * | 2019-10-10 | 2020-01-24 | 京东数字科技控股有限公司 | Format conversion method and device for system log, electronic equipment and computer medium |
| CN111324614A (en) * | 2020-03-18 | 2020-06-23 | 浩云科技股份有限公司 | Table configuration method and device, terminal equipment and storage medium |
| CN111400397A (en) * | 2020-02-29 | 2020-07-10 | 平安科技（深圳）有限公司 | Data synchronization method, device, equipment and computer storage medium |
| CN113486116A (en) * | 2021-07-07 | 2021-10-08 | 建信金融科技有限责任公司 | Data synchronization method and device, electronic equipment and computer readable medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1687927A (en) * | 2005-05-08 | 2005-10-26 | 浙江大学 | Method for implementing automatic exchange of information system data |
| US20060179070A1 (en) * | 2001-08-31 | 2006-08-10 | Bmc Software, Inc. | Service desk data transfer interface |
| CN105205117A (en) * | 2015-09-09 | 2015-12-30 | 郑州悉知信息科技股份有限公司 | Data table migrating method and device |
| CN105335412A (en) * | 2014-07-31 | 2016-02-17 | 阿里巴巴集团控股有限公司 | Method and device for data conversion and data migration |
| CN107357848A (en) * | 2017-06-27 | 2017-11-17 | 中国电子科技集团公司第二十八研究所 | Database synchronization method based on driving encapsulation |
| CN109657007A (en) * | 2018-12-17 | 2019-04-19 | 武汉达梦数据库有限公司 | Database in phase based on asynchronous submission executes method and apparatus parallel |
-
2019
- 2019-05-23 CN CN201910435627.2A patent/CN110222117A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060179070A1 (en) * | 2001-08-31 | 2006-08-10 | Bmc Software, Inc. | Service desk data transfer interface |
| CN1687927A (en) * | 2005-05-08 | 2005-10-26 | 浙江大学 | Method for implementing automatic exchange of information system data |
| CN105335412A (en) * | 2014-07-31 | 2016-02-17 | 阿里巴巴集团控股有限公司 | Method and device for data conversion and data migration |
| CN105205117A (en) * | 2015-09-09 | 2015-12-30 | 郑州悉知信息科技股份有限公司 | Data table migrating method and device |
| CN107357848A (en) * | 2017-06-27 | 2017-11-17 | 中国电子科技集团公司第二十八研究所 | Database synchronization method based on driving encapsulation |
| CN109657007A (en) * | 2018-12-17 | 2019-04-19 | 武汉达梦数据库有限公司 | Database in phase based on asynchronous submission executes method and apparatus parallel |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110659263A (en) * | 2019-09-24 | 2020-01-07 | 浪潮软件集团有限公司 | Heterogeneous database data exchange method |
| CN110727646A (en) * | 2019-10-10 | 2020-01-24 | 京东数字科技控股有限公司 | Format conversion method and device for system log, electronic equipment and computer medium |
| CN111400397A (en) * | 2020-02-29 | 2020-07-10 | 平安科技（深圳）有限公司 | Data synchronization method, device, equipment and computer storage medium |
| CN111400397B (en) * | 2020-02-29 | 2023-04-11 | 平安科技（深圳）有限公司 | Data synchronization method, device, equipment and computer storage medium |
| CN111324614A (en) * | 2020-03-18 | 2020-06-23 | 浩云科技股份有限公司 | Table configuration method and device, terminal equipment and storage medium |
| CN113486116A (en) * | 2021-07-07 | 2021-10-08 | 建信金融科技有限责任公司 | Data synchronization method and device, electronic equipment and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110222117A (en) | A kind of data conversion synchronous method, equipment and the storage medium of heterogeneous database | |
| CN101707399B (en) | Method and system for acquiring electric energy information | |
| CN103559118B (en) | A kind of method for auditing safely based on AOP and annotating information system | |
| CN102891873A (en) | Method for storing log data and log data storage system | |
| US20120173679A1 (en) | Adaptation method, adapter and adapter system for network service communication | |
| CN104516989B (en) | Incremental data supplying system and method | |
| CN107515933A (en) | Change method, server, system and the storage medium of source data database table structure | |
| CN103019874A (en) | Method and device for realizing exception handling based on data synchronization | |
| CN103246547B (en) | The method and system of remote invocation of objects | |
| CN103297485B (en) | Distributed caching automated management system and distributed caching automatic management method | |
| CN110222119A (en) | A kind of data conversion synchronous method, equipment and the storage medium of heterogeneous database | |
| CN109995643A (en) | Business mail push method and device | |
| CN101930361B (en) | Method and system for providing online data storage service | |
| EP2696297B1 (en) | System and method for generating information file based on parallel processing | |
| CN112131002A (en) | Data management method and device | |
| CN104660435A (en) | Transactional network management engine for intelligent pipeline and network management transaction processing method of transactional network management engine | |
| CN104462562A (en) | Data migration system and method based on data warehouse automation | |
| CN112422588A (en) | Internet of things equipment information processing method and system | |
| CN101763294A (en) | Method for looking up and backing up system operation log according to time slots | |
| CN103092866A (en) | Data monitoring method and monitoring device | |
| CN104836831B (en) | A kind of object method of servicing for Internet of Things | |
| CN101056210B (en) | An event processing system and method of network central management platform | |
| CN105933396B (en) | A kind of each canalization interface of collaboration realizes the method and system of long flow | |
| CN105204869B (en) | A kind of data processing method of communication system | |
| CN115617768A (en) | Log management method, system, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province Applicant after: Wuhan dream database Co., Ltd Address before: 430000 16-19 / F, building C3, future technology building, 999 Gaoxin Avenue, Donghu New Technology Development Zone, Wuhan, Hubei Province Applicant before: WUHAN DAMENG DATABASE Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Sun Feng Inventor after: Yu Yuanlan Inventor after: Zhao Jiawei Inventor before: Fu Quan Inventor before: Sun Feng Inventor before: Yu Yuanlan Inventor before: Zhao Jiawei |
|
| CB03 | Change of inventor or designer information |