CN105701147B - Configurable data shifts managing device - Google Patents
Configurable data shifts managing device Download PDFInfo
- Publication number
- CN105701147B CN105701147B CN201511009562.3A CN201511009562A CN105701147B CN 105701147 B CN105701147 B CN 105701147B CN 201511009562 A CN201511009562 A CN 201511009562A CN 105701147 B CN105701147 B CN 105701147B
- Authority
- CN
- China
- Prior art keywords
- data
- database
- managing device
- transfer
- configurable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention proposes a kind of configurable datas to shift managing device comprising: user configuration interface, for inputting setting data transfer parameters based on user and being passed to transfer driver by configuring operation therewith with trigger data transfer operation；Path management interface, the path management interface are used to provide the environment for executing the configuration operation；Driver is shifted, determines data transfer path scheme based on data transfer parameters, and the actual data transfer process of engine start is operated based on identified data transfer path scheme trigger data library therewith；Database manipulation engine is started in a manner of the specific structure and operation interface that adapt to source database and target database upon being triggered and completes the real data transfer process between source database and target database.Configurable data disclosed in this invention transfer managing device has high performance and efficiency and easy to maintain.
Description
Technical field
The present invention relates to data to shift managing device, shifts managing device more particularly, to configurable data.
Background technique
Currently, being continuously increased and becoming increasingly popular with Database Systems application, the data transfer between multiple databases
Become increasingly common and important, also, usually contains the database there are many platform in same database application system, because
This, with the data smoothing transfer techniques between/heterogeneous database be also market there is an urgent need to.
In existing technical solution, generally directed to the different types of customized operation code of database stand-alone development,
To complete transfer, duplication, extraction, filtering and the synchronization of data.
However, there are the following problems for existing technical solution: (1) since service logic and operation code are typically contaminated
Together, therefore cause exploitation and maintenance efficiency low；(2) since code logic is related to multiple line distance management and shared resource is competing
It strives, therefore causes performance loss and follow-up maintenance difficult.
Accordingly, there exist following demands: providing has high performance and efficiency and configurable data transfer pipe easy to maintain
Manage device.
Summary of the invention
In order to solve defect present in above-mentioned prior art, the invention proposes with high performance and efficiency simultaneously
Configurable data easy to maintain shifts managing device.
The purpose of the present invention is what is be achieved through the following technical solutions:
A kind of configurable data transfer managing device, the configurable data transfer managing device include:
User configuration interface, the user configuration interface are used to input setting data transfer parameters based on user and lead to therewith
It crosses configuration operation and is passed to transfer driver with trigger data transfer operation, wherein the data transfer parameters at least wrap
Include the routing information of source database and target database；
Path management interface, the path management interface are used to provide the running environment for executing the configuration operation；
Driver is shifted, transfer parameters determine data transfer path scheme to the transfer driver based on the data, and
The actual data transfer process of engine start is operated based on identified data transfer path scheme trigger data library therewith；
Database manipulation engine, the database manipulation engine upon being triggered with adapt to the source database and
The specific structure of the target database and the mode of operation interface start and complete the source database and the target data
Real data transfer process between library.
In scheme disclosed above, it is preferable that the configuration operation is by calling one or more path management API letters
Several modes is completed or by calling the configuration file based on XML format to be completed.
In scheme disclosed above, it is preferable that the source database and the target database be heterogeneous database or
Isomorphism database.
In scheme disclosed above, it is preferable that the source database and the routing information of target database limit needle
Routing policy to different condition and the additional data processing operation based on the routing policy.
In scheme disclosed above, it is preferable that the user configuration interface is the user based on file, webpage or picture and text
Interface.
In scheme disclosed above, it is preferable that the transfer driver further comprises Parameter analysis of electrochemical unit, path point
Cut unit and tables of data resolution unit, wherein the Parameter analysis of electrochemical unit is for parsing the data transfer parameters with determination
The routing policy and the additional data processing operation.The path cutting unit is for determining that data shift Working mould
Formula, wherein the data transfer operating mode includes at least the first operating mode and the second operating mode.The tables of data parsing
The unit difference of the source database and the target database on data list structure for identification.
In scheme disclosed above, it is preferable that the additional data processing operation includes one or more of following:
Data filtering, data conversion, transformation of data and secondary routing.
In scheme disclosed above, it is preferable that the tables of data resolution unit is based on the source database and mesh identified
It marks difference of the database on data list structure and determines and the field of the exact matching in tables of data is implemented to shift automatically, and logarithm
It carries out for the compatibility confirmation for having discrepant field or custom-configures according to the field prompt user of Incomplete matching in table.
In scheme disclosed above, it is preferable that first operating mode is limited for the multiple of the source database
Workflow, each of the multiple workflow are independently shared out the work path and associated therewith according to the routing policy
The additional data processing operation of connection.And each workflow is independent to read tables of data and concurrently work from source database
Make.
In scheme disclosed above, it is preferable that first operating mode is according to the restriction of scheduled different condition
Multiple workflows, i.e., when the subtask in total data transfer task meets different condition using different workflow execution numbers
According to transfer operation.
In scheme disclosed above, it is preferable that the target database includes multiple subdata bases, and according to source number
The record is routed to target subdata base according to the predetermined characteristic value and the routing policy of each record in library.
In scheme disclosed above, it is preferable that second operating mode limits unique stream that works independently, only from
Source database reads a tables of data, and then for each record successively traverse path list in read tables of data
To determine if the filter condition in each path in coupling path list, and if meeting condition, the path is used
The record is routed to target database.
In scheme disclosed above, it is preferable that the path list is pre-configured with by user.
In scheme disclosed above, it is preferable that multiple tables of data in the source database can be based on described first
Operating mode or second operating mode and be transferred or be copied in multiple tables of data in the target database.
In scheme disclosed above, it is preferable that the data filtering operation in the additional data processing operation is for true
Determine whether the record in source database is suitable for path associated with data filtering operation.
In scheme disclosed above, it is preferable that the operation of transformation of data in the additional data processing operation for pair
Record in source database carries out the deformation of predetermined way.
In scheme disclosed above, it is preferable that the transfer driver further comprises mapper, and the mapper is used
In the data in a certain path will be used to shift task according to the different mappings of the characteristic value of data record to be transferred to different
On subpath.
In scheme disclosed above, it is preferable that the transfer driver is the form of software package, can be inserted into
In application program.
Configurable data transfer managing device disclosed in this invention has the advantages that at high performance and data
Manage efficiency and easy to maintain.
Detailed description of the invention
In conjunction with attached drawing, technical characteristic of the invention and advantage will be more fully understood by those skilled in the art, in which:
Fig. 1 is the schematic diagram of the configurable data transfer managing device of embodiment according to the present invention.
Specific embodiment
Fig. 1 is the schematic diagram of the configurable data transfer managing device of embodiment according to the present invention.Such as Fig. 1 institute
Show, configurable data transfer managing device disclosed in this invention includes user configuration interface 1, path management interface 2, transfer drive
Dynamic device 3 and database manipulation engine 4.Wherein, the user configuration interface 1, which is used to input setting data based on user, shifts ginseng
Number is simultaneously passed to transfer driver 3 by configuring operation therewith with trigger data transfer operation, wherein the data transfer
Parameter includes at least the routing information of source database and target database.The path management interface 2 executes institute for providing
State the running environment of configuration operation.Transfer parameters determine data transfer path scheme to the transfer driver 3 based on the data
(illustratively comprising the work after preparation, transfer process and transfer before transfer).And therewith based on determined by
Data transfer path scheme trigger data library operation engine 4 starts actual data transfer process.The database manipulation engine 4
Upon being triggered in a manner of the specific structure and operation interface that adapt to the source database and the target database
Start and completes the real data transfer process between the source database and the target database.
Preferably, in configurable data disclosed in this invention transfer managing device, the configuration operation passes through calling
The mode of one or more path management api functions is completed or by calling the configuration file based on XML format to be completed.
Preferably, in configurable data disclosed in this invention transfer managing device, the source database and the mesh
Mark database is heterogeneous database (i.e. with the database of different structure and operation interface) or isomorphism database.
Preferably, in configurable data disclosed in this invention transfer managing device, the source database and target
The routing information of database limit for different condition routing policy (i.e. which way to go at which kind of by route) and
Additional data processing operation based on the routing policy.
Preferably, in configurable data disclosed in this invention transfer managing device, the user configuration interface 1 is base
In the user interface of file, webpage or picture and text.
Preferably, in configurable data disclosed in this invention transfer managing device, the transfer driver 3 is further
Including Parameter analysis of electrochemical unit, path cutting unit and tables of data resolution unit.Wherein, the Parameter analysis of electrochemical unit is for parsing
The data transfer parameters are with the determination routing policy and the additional data processing operation.The path cutting unit is used
In determining that data shift operating mode, wherein data transfer operating mode includes at least the first operating mode and the second work
Operation mode.The tables of data resolution unit for identification the source database and the target database on data list structure
Difference.
Preferably, in configurable data disclosed in this invention transfer managing device, the additional data processing operation
Including one or more of following: data filtering, data conversion, transformation of data and it is secondary routing (routing routed, than
As said, default direction be route to DB1/TBL, but when the triggering of some condition, just route to DB1/TBL1, and another
Condition routes to DB1/TBL2 when triggering).
Preferably, in configurable data disclosed in this invention transfer managing device, the tables of data resolution unit base
It determines in difference of the source database and target database identified on data list structure to the exact matching in tables of data
Field implements automatic transfer, and be directed to the field prompt user of Incomplete matching in tables of data and have discrepant field
Compatibility confirmation custom-configures.
Preferably, in configurable data disclosed in this invention transfer managing device, first operating mode is limited
For multiple workflows of the source database, (wherein, the workflow can be carried out successively in any suitable order, or simultaneously
Row carries out, and can be judged based on scheduled rule the starting time of each workflow), in the multiple workflow
It is each independently to be shared out the work path and additional data processing operation associated there according to the routing policy.And
Each workflow is independent to be read tables of data from source database and concurrently works.
Preferably, configurable data disclosed in this invention transfer managing device in, first operating mode according to
Scheduled different condition limits the multiple workflow, i.e., when the subtask in total data transfer task is (such as source number
According to the transfer operation of a data recording in library) when meeting different condition using different workflow execution data transfer operations
(for example, then corresponding data transfer operation is executed using workflow 1, when a certain subtask when a certain subtask meets condition 1
Meet condition 2, then executes corresponding data transfer operation using workflow 2, and so on).
Preferably, in configurable data disclosed in this invention transfer managing device, the target database includes more
A subdata base (i.e. it is the set of multiple subdata bases), and according to the predetermined characteristic value of each record in source database
And the record is routed to target subdata base by the routing policy.
Preferably, in configurable data disclosed in this invention transfer managing device, second operating mode is limited
Unique stream that works independently, only reads a tables of data from source database, and then in read tables of data
It is each to record successively traverse path list to determine if the filter condition in each path in coupling path list, and such as
Fruit meets condition, then the record being routed to target database using the path, (wherein, one-to-many data transfer mode can be with
The data transfer mode of multi-to-multi is realized by simple horizontal expansion).
Preferably, in configurable data disclosed in this invention transfer managing device, the path list is pre- by user
First configure.
Preferably, multiple in the source database in configurable data disclosed in this invention transfer managing device
Tables of data can be transferred or be copied to the target data based on first operating mode or second operating mode
In multiple tables of data in library.
Preferably, in configurable data disclosed in this invention transfer managing device, the additional data processing operation
In data filtering operate for determining whether the record in source database is suitable for road associated with data filtering operation
Diameter.
Preferably, in configurable data disclosed in this invention transfer managing device, the additional data processing operation
In transformation of data operation for carrying out deformation (such as the change of numerical value or format of predetermined way to the record in source database
Shape).
Preferably, in configurable data disclosed in this invention transfer managing device, the transfer driver 3 is further
Including mapper, the mapper will be for that will use the data transfer task in a certain path according to the spy of data record to be transferred
In the different mappings of value indicative to different subpaths.
Preferably, in configurable data disclosed in this invention transfer managing device, the transfer driver 3 is software
The form of plug-in unit can be inserted into application program.
Therefore configurable data transfer managing device disclosed in this invention has the advantage that with high property
Can and data-handling efficiency and easy to maintain.
Although the present invention is described by above-mentioned preferred embodiment, way of realization is not limited to
Above-mentioned embodiment.It will be appreciated that in the case where not departing from spirit and scope of the present invention, those skilled in the art can be with
Different change and modification are made to the present invention.
Claims (14)
1. a kind of configurable data shifts managing device, the configurable data transfer managing device includes:
User configuration interface, the user configuration interface are used to input setting data transfer parameters based on user and therewith by matching
It sets operation and is passed to transfer driver with trigger data transfer operation, wherein the data transfer parameters include at least source
The routing information of database and target database, the routing information be used for limits be directed to different condition routing policy and
Additional data processing operation based on the routing policy；
Path management interface, the path management interface are used to provide the running environment for executing the configuration operation；
Driver is shifted, transfer parameters determine data transfer path scheme to the transfer driver based on the data, and therewith
The actual data transfer process of engine start is operated based on identified data transfer path scheme trigger data library；
Database manipulation engine, the database manipulation engine is upon being triggered to adapt to the source database and described
The specific structure of target database and the mode of operation interface start and complete the source database and the target database it
Between real data transfer process；
Wherein, the transfer driver further comprises:
Parameter analysis of electrochemical unit is used to parse the data transfer parameters with the determination routing policy and the additional data
Processing operation；
Path cutting unit is used to determine that data to shift operating mode, wherein the data transfer operating mode includes at least
First operating mode and the second operating mode, first operating mode limit multiple workflows according to scheduled different condition,
So that being shifted when the subtask in total data transfer task meets different condition using different workflow execution data
It operating, second operating mode limits unique stream that works independently, only from a source database tables of data of reading, and with
Afterwards for each record in read tables of data successively traverse path list to determine if in coupling path list
The filter condition in each path, and if meeting condition, the record is routed to target database using the path；And
Tables of data resolution unit, for identification difference of the source database and the target database on data list structure
It is different.
2. configurable data according to claim 1 shifts managing device, which is characterized in that the configuration operation passes through tune
It is completed with the mode of one or more path management api functions or by calling the configuration file based on XML format complete
At.
3. configurable data according to claim 1 shifts managing device, which is characterized in that the source database and described
Target database is heterogeneous database or isomorphism database.
4. configurable data according to claim 1 shifts managing device, which is characterized in that the user configuration interface is
User interface based on file, webpage or picture and text.
5. configurable data according to claim 1 shifts managing device, which is characterized in that the additional data processing behaviour
Make to include one or more of following: data filtering, data conversion, transformation of data and secondary routing.
6. configurable data according to claim 1 shifts managing device, which is characterized in that the tables of data resolution unit
It is determined based on difference of the source database and target database identified on data list structure to the exact matching in tables of data
Field implement automatic transfer, and the field prompt user of Incomplete matching in tables of data is carried out for having discrepant field
Compatibility confirmation or custom-configure.
7. configurable data according to claim 1 shifts managing device, which is characterized in that the first operating mode limit
Surely multiple workflows of the source database are directed to, each of the multiple workflow is by independently according to the routing policy
Share out the work path and additional data processing operation associated there, and each workflow is independent from source data
It reads tables of data and concurrently works in library.
8. configurable data according to claim 1 shifts managing device, which is characterized in that the target database includes
Multiple subdata bases, and according to the predetermined characteristic value of each record in source database and the routing policy by the record
It is routed to target subdata base.
9. configurable data according to claim 1 shifts managing device, which is characterized in that the path list is by user
It is pre-configured with.
10. configurable data according to claim 1 shifts managing device, which is characterized in that in the source database
Multiple tables of data can be transferred or be copied to the target based on first operating mode or second operating mode
In multiple tables of data in database.
11. configurable data according to claim 5 shifts managing device, which is characterized in that the additional data processing
Data filtering in operation is operated for determining it is associated with data filtering operation whether the record in source database is suitable for
Path.
12. configurable data according to claim 5 shifts managing device, which is characterized in that the additional data processing
Transformation of data operation in operation is for carrying out the deformation of predetermined way to the record in source database.
13. configurable data according to claim 1 shifts managing device, which is characterized in that the transfer driver into
One step includes mapper, and the mapper will be for that will use the data transfer task in a certain path according to data record to be transferred
Characteristic value different mappings to different subpaths on.
14. configurable data according to claim 1 shifts managing device, which is characterized in that the transfer driver is
The form of software package can be inserted into application program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511009562.3A CN105701147B (en) | 2015-12-30 | 2015-12-30 | Configurable data shifts managing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511009562.3A CN105701147B (en) | 2015-12-30 | 2015-12-30 | Configurable data shifts managing device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105701147A CN105701147A (en) | 2016-06-22 |
| CN105701147B true CN105701147B (en) | 2019-11-22 |
Family
ID=56225992
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511009562.3A Active CN105701147B (en) | 2015-12-30 | 2015-12-30 | Configurable data shifts managing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105701147B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106815317A (en) * | 2016-12-23 | 2017-06-09 | 中国银联股份有限公司 | A kind of data copy method and system based on disparate databases |
| CN108984652B (en) * | 2018-06-27 | 2020-10-27 | 北京圣康汇金科技有限公司 | Configurable data cleaning system and method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101789021A (en) * | 2010-02-24 | 2010-07-28 | 浪潮通信信息系统有限公司 | Universal configurable database data migration method |
| CN104462119A (en) * | 2013-09-18 | 2015-03-25 | 腾讯科技（深圳）有限公司 | Data migration method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9996562B2 (en) * | 2013-07-09 | 2018-06-12 | Oracle International Corporation | Automated database migration architecture |
-
2015
- 2015-12-30 CN CN201511009562.3A patent/CN105701147B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101789021A (en) * | 2010-02-24 | 2010-07-28 | 浪潮通信信息系统有限公司 | Universal configurable database data migration method |
| CN104462119A (en) * | 2013-09-18 | 2015-03-25 | 腾讯科技（深圳）有限公司 | Data migration method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105701147A (en) | 2016-06-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102317535B1 (en) | Methods and systems for implementing data tracking with software development kits | |
| CN102279749B (en) | Method and equipment for merging codes | |
| CN109446218A (en) | SQL statement generation method, device and computer readable storage medium | |
| CN102081537B (en) | Method for upgrading software | |
| JP2008134705A (en) | Data processing method and data analysis device | |
| CN107239468B (en) | Task node management method and device | |
| US11853794B2 (en) | Pipeline task verification for a data processing platform | |
| Weidlich et al. | Behaviour equivalence and compatibility of business process models with complex correspondences | |
| CN105701147B (en) | Configurable data shifts managing device | |
| CN106506302B (en) | Support the communication device of dynamic MODBUS agreement mapping | |
| CN108664665A (en) | Data format method for transformation, device, equipment and readable storage medium storing program for executing | |
| CN105446757B (en) | A kind of processing method and equipment of data packet | |
| CN103810094A (en) | Executing method and device for test case and test tool | |
| CN103279347B (en) | The synchronous method of a kind of general software product line domain model and application model | |
| CN101216907A (en) | Service data processing method and its system | |
| US20230267255A1 (en) | Semiconductor equipment modeling method and device | |
| CN110442371A (en) | A kind of method, apparatus of release code, medium and computer equipment | |
| WO2015155846A1 (en) | System for storage device to execute database hash join process | |
| CN106294132B (en) | A kind of method and device managing log | |
| JP2011248852A (en) | System cooperation apparatus | |
| CN105303325B (en) | A kind of flow control method and system based on transaction scene Recognition | |
| CN113001538B (en) | Command analysis method and system | |
| CN112035169B (en) | Jump processing method and device, computer equipment and computer readable storage medium | |
| WO2018021437A1 (en) | Operation management apparatus, operation management method, and operation management system | |
| CN105607957B (en) | A kind of data acquisition analysis system based on OPC DA |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |