CN105701147A - Configurable data transfer management device - Google Patents
Configurable data transfer management device Download PDFInfo
- Publication number
- CN105701147A CN105701147A CN201511009562.3A CN201511009562A CN105701147A CN 105701147 A CN105701147 A CN 105701147A CN 201511009562 A CN201511009562 A CN 201511009562A CN 105701147 A CN105701147 A CN 105701147A
- Authority
- CN
- China
- Prior art keywords
- data
- transfer
- managing device
- database
- data transfer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a configurable data transfer management device. The configurable data transfer management device comprises a user configuration interface used for setting a data transfer parameter based on user input and accordingly transmitting the data transfer parameter to a transfer driver through configuration operation so as to trigger data transfer operation; a path management interface used for providing an environment in which the configuration operation is carried out; the transfer driver used for determining a data transfer path scheme based on the data transfer parameter and accordingly triggering a database operation engine to start a practical data transfer process based on the determined data transfer path scheme; and the database operation engine which, after being triggered, is started to finish the practical data transfer process between a source database and a target database in a way of being adaptive to specific structures and operating interfaces of both the source database and the target database. The configurable data transfer management device disclosed by the invention has high performances and high efficiency, and is easy to maintain.
Description
Technical field
The present invention relates to data transfer managing device, shift managing device more particularly, to configurable data。
Background technology
At present, being continuously increased and becoming increasingly popular along with Database Systems application, data transfer between multiple data bases becomes increasingly common and important, and, same database application system usually contains the data base of kinds of platform, therefore, with the data smoothing transfer techniques between/heterogeneous database be also market in the urgent need to。
In existing technical scheme, it is generally directed to the different types of self-defining operation code of data base's stand-alone development, to complete the transfer of data, duplication, extraction, filtration and synchronization。
But, there are the following problems for existing technical scheme: (1) is due to together with service logic is typically mingled in operation code, therefore causes that exploitation and maintenance efficiency are low；(2) relate to multiple line distance management and shared resource contention due to code logic, therefore cause performance loss and follow-up maintenance difficulty。
Accordingly, there exist following demand: provide the configurable data transfer managing device that there is high performance and efficiency and be easily maintained。
Summary of the invention
In order to solve the defect existing for above-mentioned prior art, the present invention proposes the configurable data transfer managing device having high performance and efficiency and being easily maintained。
It is an object of the invention to be achieved through the following technical solutions:
A kind of configurable data transfer managing device, described configurable data transfer managing device includes:
User configures interface, described user configures interface and is passed to transfer driver with trigger data transfer operation for inputting based on user to arrange data transfer parameters and pass through to configure operation therewith, wherein, described data transfer parameters at least includes the routing information of source database and target database；
Path management interface, described path management interface performs the running environment of described configuration operation for providing；
Transfer driver, described transfer driver determines data transfer path scheme based on described data transfer parameters, and operates, based on determined data transfer path scheme trigger data storehouse, the data transfer process that engine start is actual therewith；
Database manipulation engine, the real data transfer process that described database manipulation engine starts and completes between described source database and described target database in the way of the ad hoc structure adapting to described source database and described target database and operation interface upon being triggered。
In scheme disclosed above, it is preferable that described configuration operation is done by calling the mode of one or more path management api function or is done by calling the configuration file based on XML format。
In scheme disclosed above, it is preferable that described source database and described target database are heterogeneous database or isomorphism data base。
In scheme disclosed above, it is preferable that the routing information of described source database and target database limits the routing policy for different condition and the additional data based on this routing policy processes operation。
In scheme disclosed above, it is preferable that described user configures interface and is based on the user interface of file, webpage or picture and text。
In scheme disclosed above, preferably, described transfer driver farther includes Parameter analysis of electrochemical unit, path cutting unit and tables of data resolution unit, wherein, described Parameter analysis of electrochemical unit is used for resolving described data transfer parameters to determine that described routing policy and described additional data process operation。Described path cutting unit is used for determining data transfer mode of operation, and wherein, described data transfer mode of operation at least includes the first mode of operation and the second mode of operation。Described tables of data resolution unit is for identifying described source database and described target database difference on data list structure。
In scheme disclosed above, it is preferable that described additional data process operation include following in one or more: data filtering, data conversion, transformation of data and secondary route。
In scheme disclosed above, preferably, described tables of data resolution unit based on the source database identified and target database the difference on data list structure determine the field mated completely in tables of data implemented automatically transfer, and carry out the field prompting user of Incomplete matching in tables of data confirming or custom-configuring for having the compatible of discrepant field。
In scheme disclosed above, preferably, described first mode of operation limits for multiple workflows of described source database, and each in the plurality of workflow is shared out the work path according to described routing policy independently and additional data associated there processes operation。And each workflow is each independent to be read tables of data from source database and works concurrently。
In scheme disclosed above, it is preferable that described first mode of operation limits the plurality of workflow according to predetermined different condition, and namely the subtask in total data transfer task uses different workflow execution data transfer operation when meeting different condition。
In scheme disclosed above, it is preferable that described target database comprises multiple subdata base, and this record is routed to target subdata base by the predetermined characteristic value and described routing policy according to each record in source database。
In scheme disclosed above, preferably, described second mode of operation limits the stream that uniquely works alone, it only reads a tables of data from source database, and subsequently for each record in the tables of data read successively traverse path list to determine if the filtercondition in each path in coupling path list, if and satisfy condition, then use this path that this record is routed to target database。
In scheme disclosed above, it is preferable that described path list is pre-configured with by user。
In scheme disclosed above, it is preferable that the multiple tables of data in described source database can be transferred or be copied to based on described first mode of operation or described second mode of operation in the multiple tables of data in described target database。
In scheme disclosed above, it is preferable that whether the data filtering operation that described additional data processes in operation is applicable to operate, with this data filtering, the path being associated for the record determining in source database。
In scheme disclosed above, it is preferable that described additional data processes the transformation of data operation in operation for the record in source database carries out the deformation of predetermined way。
In scheme disclosed above, it is preferable that described transfer driver farther includes mapper, described mapper for the data transfer task by using a certain path according to the different mappings of the eigenvalue of data record to be transferred to different subpaths。
In scheme disclosed above, it is preferable that described transfer driver is the form of software package, and it can be inserted in application program。
Configurable data disclosed in this invention transfer managing device has the advantage that have high performance and data-handling efficiency and be easily maintained。
Accompanying drawing explanation
Will be more fully understood that by those skilled in the art in conjunction with accompanying drawing, the technical characteristic of the present invention and advantage, wherein:
Fig. 1 is the schematic diagram of configurable data transfer managing device according to an embodiment of the invention。
Detailed description of the invention
Fig. 1 is the schematic diagram of configurable data transfer managing device according to an embodiment of the invention。As it is shown in figure 1, configurable data disclosed in this invention transfer managing device includes user configures interface 1, path management interface 2, transfer driver 3 and database manipulation engine 4。Wherein, described user configures interface 1 and is passed to transfer driver 3 with trigger data transfer operation for inputting based on user to arrange data transfer parameters and pass through to configure operation therewith, wherein, described data transfer parameters at least includes the routing information of source database and target database。Described path management interface 2 performs the running environment of described configuration operation for providing。Described transfer driver 3 based on described data transfer parameters determine data transfer path scheme (exemplarily, it include transfer before preparation, transfer process and transfer after work)。And the data transfer process of reality is started therewith based on determined data transfer path scheme trigger data storehouse operation engine 4。The real data transfer process that described database manipulation engine 4 starts and completes between described source database and described target database in the way of the ad hoc structure adapting to described source database and described target database and operation interface upon being triggered。
Preferably, shifting in managing device at configurable data disclosed in this invention, described configuration operation is done by calling the mode of one or more path management api function or is done by calling the configuration file based on XML format。
Preferably, shifting in managing device at configurable data disclosed in this invention, described source database and described target database are heterogeneous database (namely having different structure and the data base of operation interface) or isomorphism data base。
Preferably, shifting in managing device at configurable data disclosed in this invention, the routing information of described source database and target database limits the routing policy (namely which way to go by circuit in which kind of situation) for different condition and the additional data based on this routing policy processes operation。
Preferably, shifting in managing device at configurable data disclosed in this invention, described user configures interface 1 and is based on the user interface of file, webpage or picture and text。
Preferably, shifting in managing device at configurable data disclosed in this invention, described transfer driver 3 farther includes Parameter analysis of electrochemical unit, path cutting unit and tables of data resolution unit。Wherein, described Parameter analysis of electrochemical unit is used for resolving described data transfer parameters to determine that described routing policy and described additional data process operation。Described path cutting unit is used for determining data transfer mode of operation, and wherein, described data transfer mode of operation at least includes the first mode of operation and the second mode of operation。Described tables of data resolution unit is for identifying described source database and described target database difference on data list structure。
Preferably, shift in managing device at configurable data disclosed in this invention, described additional data process operation include following in one or more: data filtering, data conversion, transformation of data and secondary route the (route namely routeing, such as, default direction is to route to DB1/TBL, but when certain condition triggers, just route to DB1/TBL1, and route to DB1/TBL2 when another one condition triggers)。
Preferably, shift in managing device at configurable data disclosed in this invention, described tables of data resolution unit based on the source database identified and target database the difference on data list structure determine the field mated completely in tables of data implemented automatically transfer, and carry out the field prompting user of Incomplete matching in tables of data confirming or custom-configuring for having the compatible of discrepant field。
Preferably, shift in managing device at configurable data disclosed in this invention, described first mode of operation limits the multiple workflows for described source database (wherein, described workflow can be sequentially carried out in any suitable order, or carry out parallel, and based on predetermined rule, the startup time of each workflow can be judged), each in the plurality of workflow is processed operation according to share out the work path and additional data associated there of described routing policy independently。And each workflow is each independent to be read tables of data from source database and works concurrently。
Preferably, shift in managing device at configurable data disclosed in this invention, described first mode of operation limits the plurality of workflow according to predetermined different condition, namely the subtask (such as the transfer operation of a data recording in source database) in total data transfer task uses different workflow execution data transfer operation (such as when meeting different condition, when a certain subtask satisfies condition 1, workflow 1 is then used to perform corresponding data transfer operation, when a certain subtask satisfies condition 2, workflow 2 is then used to perform corresponding data transfer operation, by that analogy)。
Preferably, shift in managing device at configurable data disclosed in this invention, described target database comprises multiple subdata base (namely it is the set of multiple subdata base), and this record is routed to target subdata base by the predetermined characteristic value and described routing policy according to each record in source database。
Preferably, shift in managing device at configurable data disclosed in this invention, described second mode of operation limits the stream that uniquely works alone, it only reads a tables of data from source database, and subsequently for each record in the tables of data read successively traverse path list to determine if the filtercondition in each path in coupling path list, if and satisfy condition, then use this path that this record is routed to target database (wherein, the data transfer mode of one-to-many can be expanded by simple level and realize the data transfer mode of multi-to-multi)。
Preferably, shifting in managing device at configurable data disclosed in this invention, described path list is pre-configured with by user。
Preferably, shifting in managing device at configurable data disclosed in this invention, the multiple tables of data in described source database can be transferred or be copied to based on described first mode of operation or described second mode of operation in the multiple tables of data in described target database。
Preferably, shifting in managing device at configurable data disclosed in this invention, whether the data filtering operation that described additional data processes in operation is applicable to operate, with this data filtering, the path being associated for the record determining in source database。
Preferably, shifting in managing device at configurable data disclosed in this invention, described additional data processes the transformation of data operation in operation for the record in source database carries out the deformation (deformation of such as numerical value or form) of predetermined way。
Preferably, shift in managing device at configurable data disclosed in this invention, described transfer driver 3 farther includes mapper, described mapper for the data transfer task by using a certain path according to the different mappings of the eigenvalue of data record to be transferred to different subpaths。
Preferably, shifting in managing device at configurable data disclosed in this invention, described transfer driver 3 is the form of software package, and it can be inserted in application program。
Therefore, configurable data disclosed in this invention transfer managing device has the advantage that have high performance and data-handling efficiency and be easily maintained。
Although the present invention is described by above-mentioned preferred implementation, but its way of realization is not limited to above-mentioned embodiment。It will be appreciated that when without departing from present subject matter and scope, the present invention can be made different changing and modifications by those skilled in the art。
Claims (18)
1. a configurable data transfer managing device, described configurable data transfer managing device includes:
User configures interface, described user configures interface and is passed to transfer driver with trigger data transfer operation for inputting based on user to arrange data transfer parameters and pass through to configure operation therewith, wherein, described data transfer parameters at least includes the routing information of source database and target database；
Path management interface, described path management interface performs the running environment of described configuration operation for providing；
Transfer driver, described transfer driver determines data transfer path scheme based on described data transfer parameters, and operates, based on determined data transfer path scheme trigger data storehouse, the data transfer process that engine start is actual therewith；
Database manipulation engine, the real data transfer process that described database manipulation engine starts and completes between described source database and described target database in the way of the ad hoc structure adapting to described source database and described target database and operation interface upon being triggered。
2. configurable data according to claim 1 transfer managing device, it is characterised in that described configuration operation is done by calling the mode of one or more path management api function or is done by calling the configuration file based on XML format。
3. configurable data according to claim 2 transfer managing device, it is characterised in that described source database and described target database are heterogeneous database or isomorphism data base。
4. configurable data according to claim 3 transfer managing device, it is characterised in that the routing information of described source database and target database limits the routing policy for different condition and the additional data based on this routing policy processes operation。
5. configurable data according to claim 4 transfer managing device, it is characterised in that described user configures interface and is based on the user interface of file, webpage or picture and text。
6. configurable data according to claim 5 transfer managing device, it is characterized in that, described transfer driver farther includes Parameter analysis of electrochemical unit, path cutting unit and tables of data resolution unit, wherein, described Parameter analysis of electrochemical unit is used for resolving described data transfer parameters to determine that described routing policy and described additional data process operation, described path cutting unit is used for determining data transfer mode of operation, wherein, described data transfer mode of operation at least includes the first mode of operation and the second mode of operation, described tables of data resolution unit is for identifying described source database and described target database difference on data list structure。
7. configurable data according to claim 6 transfer managing device, it is characterised in that described additional data process operation include following in one or more: data filtering, data conversion, transformation of data and secondary route。
8. configurable data according to claim 7 transfer managing device, it is characterized in that, described tables of data resolution unit based on the source database identified and target database the difference on data list structure determine the field mated completely in tables of data implemented automatically transfer, and carry out the field prompting user of Incomplete matching in tables of data confirming or custom-configuring for having the compatible of discrepant field。
9. configurable data according to claim 8 transfer managing device, it is characterized in that, described first mode of operation limits the multiple workflows for described source database, each in the plurality of workflow is processed operation according to share out the work path and additional data associated there of described routing policy independently, and each workflow is each independent reads tables of data from source database and work concurrently。
10. configurable data according to claim 9 transfer managing device, it is characterized in that, described first mode of operation limits the plurality of workflow according to predetermined different condition, and namely the subtask in total data transfer task uses different workflow execution data transfer operation when meeting different condition。
11. configurable data according to claim 10 transfer managing device, it is characterized in that, described target database comprises multiple subdata base, and this record is routed to target subdata base by the predetermined characteristic value and described routing policy according to each record in source database。
12. configurable data according to claim 11 transfer managing device, it is characterized in that, described second mode of operation limits the stream that uniquely works alone, it only reads a tables of data from source database, and subsequently for each record in the tables of data read successively traverse path list to determine if the filtercondition in each path in coupling path list, if and satisfy condition, then use this path that this record is routed to target database。
13. configurable data according to claim 12 transfer managing device, it is characterised in that described path list is pre-configured with by user。
14. configurable data according to claim 13 transfer managing device, it is characterized in that, the multiple tables of data in described source database can be transferred or be copied to based on described first mode of operation or described second mode of operation in the multiple tables of data in described target database。
15. configurable data according to claim 14 transfer managing device, it is characterised in that whether the data filtering operation that described additional data processes in operation is applicable to operate, with this data filtering, the path being associated for the record determining in source database。
16. configurable data according to claim 15 transfer managing device, it is characterised in that described additional data processes the transformation of data operation in operation for the record in source database carries out the deformation of predetermined way。
17. configurable data according to claim 16 transfer managing device, it is characterized in that, described transfer driver farther includes mapper, described mapper for the data transfer task by using a certain path according to the different mappings of the eigenvalue of data record to be transferred to different subpaths。
18. configurable data according to claim 17 transfer managing device, it is characterised in that described transfer driver is the form of software package, and it can be inserted in application program。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511009562.3A CN105701147B (en) | 2015-12-30 | 2015-12-30 | Configurable data shifts managing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511009562.3A CN105701147B (en) | 2015-12-30 | 2015-12-30 | Configurable data shifts managing device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105701147A true CN105701147A (en) | 2016-06-22 |
| CN105701147B CN105701147B (en) | 2019-11-22 |
Family
ID=56225992
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511009562.3A Active CN105701147B (en) | 2015-12-30 | 2015-12-30 | Configurable data shifts managing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105701147B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106815317A (en) * | 2016-12-23 | 2017-06-09 | 中国银联股份有限公司 | A kind of data copy method and system based on disparate databases |
| CN108984652A (en) * | 2018-06-27 | 2018-12-11 | 北京圣康汇金科技有限公司 | A kind of configurable data cleaning system and method |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101789021A (en) * | 2010-02-24 | 2010-07-28 | 浪潮通信信息系统有限公司 | Universal configurable database data migration method |
| US20150019488A1 (en) * | 2013-07-09 | 2015-01-15 | Oracle International Corporation | Automated database migration architecture |
| CN104462119A (en) * | 2013-09-18 | 2015-03-25 | 腾讯科技（深圳）有限公司 | Data migration method and device |
-
2015
- 2015-12-30 CN CN201511009562.3A patent/CN105701147B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101789021A (en) * | 2010-02-24 | 2010-07-28 | 浪潮通信信息系统有限公司 | Universal configurable database data migration method |
| US20150019488A1 (en) * | 2013-07-09 | 2015-01-15 | Oracle International Corporation | Automated database migration architecture |
| CN104462119A (en) * | 2013-09-18 | 2015-03-25 | 腾讯科技（深圳）有限公司 | Data migration method and device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106815317A (en) * | 2016-12-23 | 2017-06-09 | 中国银联股份有限公司 | A kind of data copy method and system based on disparate databases |
| CN108984652A (en) * | 2018-06-27 | 2018-12-11 | 北京圣康汇金科技有限公司 | A kind of configurable data cleaning system and method |
| CN108984652B (en) * | 2018-06-27 | 2020-10-27 | 北京圣康汇金科技有限公司 | Configurable data cleaning system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105701147B (en) | 2019-11-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102637214B (en) | Method and system for synchronizing general data among database services | |
| US20160352578A1 (en) | System and method for adaptive paths locator for virtual network function links | |
| US20190068454A1 (en) | Device model to thing model mapping | |
| US9160697B2 (en) | Data delivery optimization | |
| JP2017520214A (en) | Method, apparatus and system for flow entry configuration | |
| CN104145257A (en) | Semantic cache cloud services for connected devices | |
| CN107239468B (en) | Task node management method and device | |
| US9674053B2 (en) | Automatic target selection | |
| EP2961108A1 (en) | Switch and program | |
| US20160080462A1 (en) | Data processing method and apparatus in service-oriented architecture system, and the service-oriented architecture system | |
| CN106506302B (en) | Support the communication device of dynamic MODBUS agreement mapping | |
| US20190312611A1 (en) | File transmission/reception device and control method of file transmission/reception device | |
| CN105069059A (en) | Database synchronization method and apparatus | |
| JP2017069895A (en) | Fault separation method and administrative server for performing fault separation | |
| JP5971057B2 (en) | Storage device, setting method and setting program | |
| CN105701147A (en) | Configurable data transfer management device | |
| CN106327143B (en) | Method and apparatus for nodes in a flow | |
| CN103279347A (en) | Universal software product line domain model and application model synchronization method | |
| CN108574719B (en) | Cloud host creation method and device | |
| CN102457559B (en) | Transaction processing method by using server cluster | |
| CN109450798A (en) | The management method and computer readable storage medium of routing table information | |
| CN109213518A (en) | A method of the system automation based on git persistently constructs | |
| CN103914304A (en) | Method for converting different structure type parameters on basis of SAP (service access point) platforms | |
| CN111382983B (en) | Workflow control method, workflow node and system | |
| WO2014136531A1 (en) | Network system, node, network management method, and computer-readable recording medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |